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Key findings 

This analysis of response data from middle school 
students and teachers on the California School 
Climate, Health, and Learning Survey examines the 
appropriateness of the survey for identifying needs 
and monitoring changes in school climate. The study 
finds that student and staff surveys validly and reliably 
assess distinct school climate domains, such as 
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Summary 


A growing number of states and school districts use school climate assessments in prog- 
ress reporting systems (Nathanson, McCormick, Kemple, & Sypek, 2013; Shah, 2013) and 
are interested in incorporating these assessments into accountability systems (Rockoff & 
Turner, 2010). This study examines the appropriateness of the California School Climate, 
Health, and Learning Survey (CahSCHLS) for identifying needs and monitoring improve' 
ments in school climate domains and for differentiating between individuals in their pen 
ceptions of school climate and between schools in their levels of school climate. It analyzes 
survey response data for 730,160 grade 7 students from 2004/05 through 2010/11 in 1,117 
California middle schools and 16,255 teachers from 2008/09 through 2010/11 in 812 Cali- 
fornia middle schools to examine the following research questions: 

• What school climate domains in middle school do the survey items measure? 

• How reliably do the survey items measure school climate at the respondent and 
school levels? 

• How stable are the schooHevel climate measures? 

• Are the school climate measures related to student achievement and discipline 
infractions? 

Three analytic strategies were used: exploratory/confirmatory factor analysis, estimation 
of respondent- and schooHevel reliability, and estimation of correlations of school climate 
measures over time and with student academic performance and suspensions. 

School climate domains measured by surveys 

Analyses of responses to the 40 student survey questions relevant to students’ in-school 
experiences find that this survey can be used to validly and reliably assess student percep- 
tions about six school climate domains: 

• Safety and connectedness. 

• Caring relationships with adults. 

• Meaningful participation. 

• Substance use at school. 

• Bullying and discrimination. 

• Delinquency. 

Analysis of teacher responses to the 64 staff survey questions finds that this survey can be 
used to validly and reliably measure teacher perceptions of seven school climate domains: 

• Support and safety. 

• Caring staff-student relationships. 

• Staff-peer relationships. 

• Professional development needs. 

• Student health and engagement. 

• Student delinquency. 

• Resource provision. 

For both student and staff surveys the identified school climate domains are consistent 
with those identified by other middle school climate surveys. 


Reliability of school climate measures 


All the school climate measures exhibit adequate respondent- and school-level reliabili- 
ty, indicating that the survey items measuring school climate at the respondent level do 
so consistently and can be used to identify differences in average school climate across 
schools. To obtain reliable school-level climate scores, a school must sample at least 100 
students and 10 teachers. 

Stability of school climate measures 

School-level scores were more stable across one-year intervals for the staff survey than for 
the student survey. The school-level scores based on the student survey were slightly less 
stable than the scores in another school climate survey (Brand, Felner, Shim, Seitsinger, & 
Dumas, 2003), while those based on the teacher responses on the staff survey were slightly 
more stable than found in a previous survey (Brand, Felner, Seitsinger, Burns, & Bolton, 
2008). The school-level correlations were weaker for the two-year intervals than for the 
one-year intervals for all of the student survey measures and for six of the seven staff survey 
measures. However, the analytic sample of schools that administered the surveys in con- 
secutive years is small and unlikely to be representative of schools in the state, so compar- 
isons between the one- and two-year stability estimates should be interpreted cautiously. 

Predictive nature of school climate measures 

All the school-level school climate measures were associated in expected ways with student 
academic performance and suspensions. Student performance was higher and suspension 
rates were lower in schools with a positive school climate. These results support the valid- 
ity of the survey measures for each of the school climate domains identified in the study. 
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Why this study? 


A troubling number of schools and districts are stuck in a pattern of high dropout rates, 
high teacher turnover, poor student engagement and attendance, and low student profb 
ciency in core subjects. One thing many of these schools have in common is a negative 
school climate, which increasingly is recognized as a major barrier to success (for example, 
Bryk, Sebring, Allensworth, Luppesco, & Easton, 2010). A growing body of research sug- 
gests that students who attend schools with a positive school climate — schools that are 
safe and supportive and that provide opportunities for students to engage in relevant and 
interesting activities — have better attendance, academic performance, and academic moti- 
vation and stronger socioemotional competence (Anderson, 1982; Benninga, Berkowitz, 
Kuehn, & Smith, 2003; Blum, McNeely, & Rinehart, 2002; Bowen & Bowen, 1999; Dejung 
& Duckworth, 1986; Eccles et al., 1993; Hanson, Austin, & Lee-Bayha, 2004; Rumberger, 
1987; Voight, Austin, & Hanson, 2013; Wu, Pink, Crain, & Moles, 1982). 

School climate also affects teachers. A good school climate is significantly related to lower 
teacher burnout (Grayson & Alvarez, 2008), greater teacher job satisfaction, and higher 
teacher retention (Kelly, 2004; Loeb, Darling-Hammond, & Luczak, 2005). Positive teacher 
perceptions of school climate are also associated with higher productivity (Bevans, Brad- 
shaw, Miech, & Leaf, 2007; Lee, Dedrick, & Smith, 1991; Taylor & Tashakkori, 1995), 
improved teacher efficacy (Bevans et al., 2007) and greater fidelity in implementing new 
curricula and interventions (Beets et al, 2008; Gregory, Henry, & Schoeny, 2007). 

Problems related to school climate are particularly acute in middle schools. The rate of violent 
incidents is almost twice as high in middle schools as in elementary and high schools (Neiman, 
2011). Students in middle schools also report substantially higher bullying rates (Neiman, 2011). 

Improving school climate is important for its intrinsic value (a safe and supportive environ- 
ment is good in and of itself) and for its instrumental value in improving student academic 
performance and college and career readiness (Farrington et al., 2012). School climate has 
promise as a focus of intervention because it incorporates some of the root factors that 
undergird student academic performance (Bryk et al., 2010). Whereas schools and districts 
may not be able to intervene in other areas affecting student performance, such as fami- 
lies, neighborhoods, and macrosocial policy environments, school climate is within their 
ability to influence (Bradshaw, Koth, Thornton, & Leaf, 2009). 

Understanding school climate 

Getting an accurate picture of school climate is a crucial first step. Many state education 
agencies are developing school climate policies that include school climate assessments and 
data-driven improvement plans in schools (Piscatelli & Lee, 2011). The U.S. Department of 
Education has funded recent initiatives to support state education agency efforts to priori- 
tize school climate, including the Safe and Supportive Schools initiative, the Safe Schools- 
Healthy Students initiative, and the Promise Neighborhood initiative. Grantees must 
measure, monitor, and evaluate school climate-related indicators. The California Office to 
Reform Education (CORE) — a consortium of 10 of the largest local education agencies in 
California — has proposed building school climate measures into the school accountability 
system (CORE, 2013), which would place school climate alongside standardized test scores 
as a component of high-stakes accountability. All these efforts require assessments with 
proven validity for interpreting scores as measures of school climate. 
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Understanding school climate has been the primary focus of the Middle Grades School 
Climate Alliance at the Regional Educational Laboratory West. The alliance operates as a 
networked improvement community in which each participating school convenes a team 
to analyze its school safety and climate data as a first step in selecting, implementing, and 
piloting strategies for creating a more positive school climate. The alliance’s California 
members have been using the state department of education’s California School Climate, 
Health, and Learning Survey (CahSCHLS) system, which consists of three interrelated 
surveys — one for students, one for school staff, and one for parents — for use in elemen- 
tary, middle, and high schools (box 1). These surveys address particular school climate 
domains, such as relationships, academic expectations, and physical safety. (See appendix 
A for more detailed information on school climate domains and how these domains relate 
to student learning.) 

Research questions 

For this report, intended for researchers, data users, and evaluation personnel in educa- 
tion agencies, the study team analyzed Cal-SCHLS data from California middle schools to 
identify the school climate domains included in the survey; assess the reliability, stability, 
and accuracy of the surveys’ measures for these domains; and determine how well perfor- 
mance in specific domains predicts other school-level academic and behavioral outcomes. 
Specifically, the study considered four research questions: 

1. What school climate domains in middle school do the survey items measure? 

This question concerns the usefulness of the surveys for identifying specific school climate- 
related needs at a school and for monitoring progress in improving specific school climate 
domains. 

2. How reliably do the survey items measure school climate at the respondent and 
school levels? 

This question seeks to identify how consistently the survey items measure each school 
climate domain and how well the school climate measures differentiate between schools. 
This question also considers how many students and teachers must be surveyed to obtain 
reliable scores for individual schools. 

3. How stable are the school-level climate measures? 

This question helps survey users understand which of the identified school climate domains 
are persistent and which are transitory features of a school’s environment. The results on 
stability also provide lower bound estimates of test-retest reliability, which can help in 
interpreting the size of observed changes in school climate scores in a given school. 

4. Are the school climate measures related to student achievement and discipline 
infractions? 

This question explores the relationship between school climate measures and other mea- 
sures of school functioning — in particular, academic achievement and suspensions. 
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Box 1. The California School Climate, Health, and Learning Survey system 

This study examines the psychometric properties of one school climate instrument — the middle 
school surveys of the California School Climate, Health, and Learning Survey (Cal-SCHLS) 
system. Developed collaboratively by the California Department of Education and WestEd, the 
system consists of surveys for students, school staff, and parents for elementary, middle, and 
high schools. 

All California districts that received Title IV (The Safe and Drug Free Schools and Communi- 
ties Act) funds from 2003/04 to 2010/11 were required to administer the Cal-SCHLS student 
and staff survey every two years. Only about 10 percent of districts — those that did not accept 
Title IV funds — did not administer the survey. Because of funding constraints, administration 
of Cal-SCHLS became voluntary in 2010/11. Nonetheless, 80 percent of districts continued to 
administer the surveys. 

Most of the analyses for this study were based on data collected during 2008/09 and 
2009/10, the last two-year period that survey administration was mandatory. The stability 
analyses, however, were based on student data collected between 2004/05 and 2010/11 
and teacher data collected between 2008/09 and 2010/11. No changes in survey questions 
occurred during the years covered in the analytic samples. 

The student survey. The Cal-SCHLS middle school student survey includes a core module and 
supplementary modules targeting specific themes. The core module includes 115 items that 
assess students’ perceptions of their school environment and their personal health and risk 
behaviors. The module was designed to assess school connectedness, developmental sup- 
ports at the school (caring relationships, high expectations, and opportunities for meaningful 
participation), and school violence and safety. Only core module items that are relevant to 
students’ in-school experiences are included in this study. The 40 items that apply to in-school 
experiences are shown in table B1 in appendix B. 

The staff survey. The Cal-SCHLS middle school staff survey consists of three sections: the first 
for all staff and the other two for specialized staff. The study used only the first section and 
analyzed only the responses from teachers, not those from other school staff. That section 
includes 69 items designed to assess five school climate areas: safe learning environments, 
norms and standards that encourage student learning, quality of relationships between staff 
and between staff and students, student behaviors that facilitate learning, and services and 
programs that address nonacademic barriers to learning (Austin & Bailey, 2008). All items from 
the staff survey are shown in table B2 in appendix B, except for five items on staff background. 

Many of the student and staff survey questions were designed to measure the same 
school climate concept from two different perspectives — that of the student and that of the 
teacher. For example, on caring relationships between staff and students, the student survey 
asks, “At this school there is an adult who really cares about me," while the staff survey asks, 
“How many adults in this school really care about every student?” 
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Box 2. Study methodology 


Data 

School climate data. The analytic student sample consists of 730,160 grade 7 students in 
1,117 California middle schools surveyed from 2004/05 through 2010/11. Because only 
grade 7 students were surveyed, different grade 7 students were surveyed across years. The 
teacher sample consists of 16,255 grade 6-8 teachers in 812 middle schools surveyed from 
2008/09 through 2010/11. Because teachers tend to remain at the same school for some 
time, the same teachers were likely surveyed year after year, although the data’s anonymity 
prevents confirming that. Only data from 2008/09 and 2009/10 were used to answer research 
questions 1, 2, and 4 because this is the most recent period in which the participating schools 
and districts were representative of the state’s schools and districts. 

School academic performance data. To examine the relationship between school climate and 
academic performance (research question 4), the study team analyzed school performance 
data from the California Department of Education’s Standardized Testing and Reporting 
program using California Standards Test scores for grade 7 and 8 English language arts and 
grade 7 math. The test is criterion-referenced to state-adopted academic content standards. 
Academic performance is based on test score data aggregated at the school level. 

Suspension data. School-level suspension and expulsion data were from the California Depart- 
ment of Education’s Uniform Management Information Reporting System database, which 
includes the number of students suspended and expelled, by infraction, for each school in 
the state. The California Department of Education does not provide suspension and expulsion 
data for specific grades. Suspension and expulsion counts were converted into rates per 100 
students. Expulsion rates, however, did not vary enough across schools to be included in the 
analyses. Researchers then examined the relationship of school-aggregate construct scores 
with overall school suspension rates. 

Analysis 

Separate analyses were conducted for the student survey and the staff survey. (See appendix C 
for a more detailed description of the analytic methods and appendix D for more detailed results.) 

What school climate domains in middle school do the survey items measure? Researchers 
estimated a series of exploratory and confirmatory factor analysis models to determine how 
well a set of unobserved variables, or factors, explain the survey response patterns. Factor 
analysis techniques were used to determine the number of underlying school climate factors 
measured by the Cal-SCHLS survey items and to understand what those underlying factors are. 

Data were analyzed for 214,465 grade 7 students in 997 middle schools that adminis- 
tered the student survey and 14,806 teachers in 794 middle schools that administered the 
staff survey in the 2008/09 or 2009/10 academic years — the most recent period in which the 
participating schools and districts were representative of California schools and districts. 

After the survey items were mapped onto the underlying school climate domains, 
respondent-level summary measures of each domain were calculated by averaging ratings on 
the associated items. School-level summary scores were created by averaging respondents’ 
summary measures for each school. These school-level summary scores served as the basis 
of the stability and validity analyses below. 

How reliably do the survey items measure school climate at the respondent and school 
levels? Respondent-level internal consistency reliability estimates using Cronbach’s alpha were 

(continued) 
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Box 2. Study methodology (continued) 


calculated to assess the extent to which each survey item used in a scale produces similar 
scores (the degree to which the scores produced by the items are internally consistent). The 
study team used Nunnaly’s (1978) criterion of .70 as the cutoff for determining acceptable 
internal consistency reliability for the scales. 

To determine school-level reliability, the intraclass correlation (the ratio of the between- 
school variation over the total variation of a measure) was used to estimate the proportion of 
the total variance of each school climate scale located between rather than within schools. The 
intraclass correlation is incorporated into the Spearman-Brown prediction formula to compute 
school-level reliabilities (r) for each scale (Raudenbush & Bryk, 2001; Winer, Brown, & Michels, 
1991). The Fastenau, Bennett, and Denburg (1996) criterion of .60 was used as the threshold 
for acceptable school-level reliability to interpret r coefficients for interrater reliability. Because 
the reliability of the aggregated measures is strictly a function of the intraciass correlation and 
the number of survey respondents within each school, the report presents a range of reliability 
values corresponding to different school sample sizes. The data sources and analytic sample 
used to examine this research question are the same as those used to address the first one. 

How stable are the school-level climate measures? Individual school climate scores were 
aggregated to the school level to compute average correlations of school-level scores over 
one- and two-year intervals. Because of changes in school leadership and other factors that 
affect the school environment over one- and two-year periods, it is difficult to ascertain whether 
observed changes in scores are due to measurement instability or to actual change. However, 
because changes over a one-year period would reflect both measurement instability and 
actual change during that year, correlations between scores over a one-year period are likely 
to provide a lower bound estimate of test-retest reliability. The examination and description of 
climate score stability also provide a benchmark for interpreting the magnitude of observed 
changes in school climate scores in specific middle schools. 

Between 2004/05 and 2010/11 there were 255 instances (among 129 schools) in which 
the same school administered the student survey in consecutive years and 1,941 instances 
(among 959 schools) in which the same school administered the student survey at a two-year 
interval. The aggregated student survey school climate measures were based on the respons- 
es of different grade 7 students across years. Between 2008/09 and 2010/11 there were 
42 instances (among 34 schools) in which a school administered the staff survey in consec- 
utive years and 119 instances (among 119 schools) in which a school administered the staff 
survey in a two-year interval. Thus, for both the student and teacher data, the analytic sample 
of schools that administered the surveys in consecutive years is small and unlikely to be rep- 
resentative of schools in the state, so comparisons between the one- and two-year stability 
estimates should be interpreted cautiously. 

Are the school climate measures related to student achievement and discipline infractions? 

Using data collected in 2008/09 and 2009/10 the study team estimated correlations between 
school-level climate scores and school-level measures of academic achievement and suspen- 
sions. Based on previous research (Biddle, 2005), the team assumed that correlations of .30 
or higher provide evidence of predictive validity. 

Sensitivity analyses. Student and staff survey response rates vary substantially across schools. 
Because the risk of biased results rises as the response rate drops, analyses were conducted 
to determine whether schools with low student or teacher response rates differed from other 
schools in climate, staff, student demographic, or other characteristics. The team also conduct- 
ed sensitivity analyses to determine whether low response rates biased the results. 
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The study relied on data from California public schools’ administration of the Cal-SCHLS 
student and staff survey instruments, California Standards Test scores for grade 7 and 8 
English language arts and grade 7 math, and schooUevel suspension and expulsion data 
from the California Department of Education’s Uniform Management Information Report' 
ing System database (see box 2 and appendix C for details on data and methods). 

Study findings 


Overall, the analysis showed that the 40 questions on the CahSCHLS student survey that 
are relevant to students’ in-school experiences can be used to measure six school climate 
domains reliably and validly: safety and connectedness, caring relationships with adults, 
meaningful participation, substance use at school, bullying and discrimination, and delin- 
quency. The internal consistency reliability of these six constructs ranged from .68 to .88, 
and the reliabilities were consistent with those from studies using other school climate 
instruments (Voight & Hanson, 2012). At the school level — the level at which a school’s 
overall climate is measured — the measure for each domain exhibited reliabilities above .80 
for sample sizes of approximately 200 students or more. 

The schooUevel scores were slightly less stable over time than those from another school 
climate survey (Brand et al., 2003). For the CahSCHLS student survey, scores on five of 
the six measures exhibited a correlation of .49 or higher across one-year intervals and .41 
or higher across two-year intervals. Substance use at school was less stable over time than 
the other measures. Predictive validity was supported for all student-reported measures 
except bullying and discrimination. All the school-level measures are associated in expect- 
ed ways with student academic performance and suspensions (indicators of positive aspects 
of school climate, such as caring relationships with adults, were associated with higher 
student academic performance and lower school suspension rates). 

The 64 questionnaire items on the Cal-SCHLS staff survey measure seven school climate 
domains: support and safety, caring staff-student relationships, staff-peer relationships, 
professional development needs, student health and engagement, student delinquency, and 
resource provision. The professional development needs domain had poor reliability (.50) 
at the school level and, in stability analyses, evinced a correlation of -.07 across consecu- 
tive years. The internal consistency reliability of the six other domains all exceeded .82. 
In most cases these constructs’ reliabilities are higher than those of other school climate 
surveys (Voight & Hanson, 2012). Each teacher-reported school climate measure exhib- 
ited correlations of .56 or higher across one-year intervals and .38 or higher across two- 
year intervals, and each was associated in expected ways with student achievement and 
suspensions. 

A more detailed description of the results is presented below. (See appendixes C and D for 
a more detailed description of the models, analysis, and results.) 

What school climate domains in middle school do the survey items measure? 

Student survey. The Cal-SCHLS student survey measures six school climate domains: 
safety and connectedness, caring relationships with adults, meaningful participation, 
substance use at school, bullying and discrimination, and delinquency (figure 1). These 
domains are consistent with those identified in other middle school climate surveys 
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(Voight & Hanson, 2012); however, the CahSCHLS does not assess some domains that are 
measured in other middle school student surveys: classroom order and fairness, community 
involvement, administrative leadership, parent involvement, physical environment, peer 
relationships, and respect for diversity. 1 

Other findings include the following: 

• Two CahSCHLS items that ask about students’ perceptions of safety at school (for 
example, “I feel safe in my school”) appear to assess the same domain as four ques- 
tions asking about school connectedness (for example, “I feel close to people at 
this school”). The study team labeled this domain “safety and connectedness.” 

• Consistent with prior psychometric analyses on this same survey (Hanson & Kim, 
2007), six questions asking about caring relationships with adults at school and 
high-expectation messages from adults at school assess teacher-student supportive 
relationships (caring relationships with adults). 

• The data revealed that the survey assesses the three distinct domains of student 
substance use at school, bullying and harassment victimization, and delinquency. 
Questions that ask about being victimized by physical violence (for example, being 
pushed or shoved), about nonphysical aggression (for example, mean rumors or 
lies spread), and about being harassed or bullied because of gender, race/ethnicity, 
religion, or other status all measure a single dimension of victimization. 

• The school climate domains measured by the student survey cover two of the three 
main categories in the federal Safe and Supportive Schools model — engagement 
and safety (see figure A1 in appendix A). None of the identified domains corre- 
spond to the environment domain. 

Staff survey. Teacher responses on the staff survey measure seven school climate domains: 
support and safety, caring staff-student relationships, staff-peer relationships, professional 
development needs, student health and engagement, student delinquency, and resource 
provision (figure 2). 

• Many of the 25 items that measure support and safety were intended to measure 
multiple distinct school climate domains, such as positive student learning envi- 
ronment (items 6-11), meaningful participation (items 16-20), cultural sensitivity 
(items 21-25), clarity of discipline policies (items 26-28), and perceived school 
safety (items 29 and 30). However, these more specific factors were too strongly 
correlated with each other to identify empirically distinct measures. 

• Similar to the student survey, five staff survey questions that ask about caring 
relationships with students and about high expectations for students assess caring 
staff-student relationships. Two questions that ask about fair treatment of students 
also assess this domain. 

Overlap of domains assessed by student and staff survey. Table 1 shows how the 
domains identified from the analysis of staff and student survey data overlap so that 
inconsistencies and consistencies in areas assessed by the two surveys can be more easily 
seen. The staff survey domains of support and safety and student health and engagement 
are represented by the student survey domains of safety and connectedness and mean- 
ingful participation. On the staff survey, student delinquency encompasses three of the 
domains identified using the student survey: substance use at school, bullying and dis- 
crimination, and delinquency. Both surveys had a distinct measure of caring relationships 
between students and staff. 


Teacher responses 
on the staff 
survey measure 
seven school 
climate domains 
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Figure 1. School climate domains on the student survey of the California School Climate, Health, and 
Learning Survey, grade 7 students 


Domain 


Survey item 



I feel close to people at this school. 

I am happy to be at this school. 

I feel like I am a part of this school. 
Teachers at school treat students fairly. 
I feel safe in my school. 

How safe do you feel at school? 


At my school, there is an adult who ... 

... really cares about me. 

... tells me when I do a good job. 

... notices when I am not there. 

... always wants me to do my best. 

... listens to me when I have something to say. 
... believes I will be a success. 


At school, I ... 

... do interesting activities. 

... help decide things like class activities or rules. 
... do things that make a difference. 


Past 30 days on school property 
... smoked cigarettes. 

... had at least one drink of alcohol. 
... smoked marijuana. 

... used any other illegal drug. 


Past 12 months on school property 
... been pushed, shoved, or slapped. 

... been afraid of being beaten up. 

... had mean rumors or lies spread about you. 

... had sexual jokes, comments, or gestures made to you. 

... been made fun of because of your looks or the way you talk. 
... had your property stolen or deliberately damaged. 

... had mean rumors or lies spread about you on the Internet. 
... harassed or bullied — gender. 

... harassed or bullied — race/ethnicity. 

... harassed or bullied — religion. 

... harassed or bullied — gay/lesbian. 

... harassed or bullied — physical or mental disability. 

... harassed or bullied — for any other reason. 


Past 12 months on school property 
... been in a physical fight. 

... been offered, sold, or given illegal drug. 

... damaged school property on purpose. 

... carried a gun. 

... carried any other weapon. 

... been threatened or injured with a weapon. 

... seen someone carrying a gun, knife, or other weapon. 


Source: Adapted with permission from California School Climate, Health, and Learning Survey [middle school student survey], by WestEd, 2010. 
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Figure 2. School climate domains on the staff survey of the California School Climate, Health, and 
Learning Survey, middle school teachers 

Domain Survey item 



This school ... 

... is a supportive place for students to learn. 

... sets high standards for academic performance. 

... promotes academic success for all students. 

... emphasizes helping students academically. 

... provides adequate support services for students. 

... emphasizes teaching lessons in relevant ways. 

... is a supportive and inviting place for staff to work. 

... promotes trust and collegiality among staff. 

... provides opportunities for student decisionmaking. 

... gives students class discussion opportunities. 

... gives students extracurricular activity opportunities. 
... gives students opportunities to make a difference. 

... encourages students to enroll in rigorous courses. 

... emphasizes culturally relevant instruction. 

... has staff examine their cultural biases through P.D. 
... prioritizes closing the racial achievement gap. 

... fosters appreciation for student diversity. 

... emphasizes showing respect for student cultures. 

... clearly communicates rules to students. 

... handles discipline problems fairly. 

... effectively handles discipline and behavior problems. 
... is a safe place for students. 

... is a safe place for staff. 

...is welcoming to and facilitates parent involvement. 

... has clean and well-maintained facilities. 


Adults in this school ... 

... really care about every student. 

... acknowledge and pay attention to students. 
... want every student to do their best. 

... listen to what students have to say. 

... believe that every student can be a success. 
... treat all students fairly. 

... treat every student with respect. 


Adults in this school ... 

... have close professional relationships with each other. 
... support and treat each other with respect. 

... feel a responsibility to improve this school. 


I need training in ... 

... meeting academic standards. 

... evidence-based methods of instruction. 

... positive behavioral supports/classroom management. 
... working with diverse racial and cultural groups. 

... culturally relevant pedagogy for students. 

... serving English language learners. 

... closing the achievement gap. 

... serving special education students. 

... meeting the social and emotional needs of students. 
... creating a positive school environment. 


Students at this school ... 

... are healthy and physically fit. 

... arrive at school alert and rested. 
... are motivated to learn. 

... are well-behaved. 


How much of a problem at this school is (reverse-coded) 
... student alcohol and drug use. 

... student tobacco use. 

... harassment and bullying among students. 

... physical fighting between students. 

... disruptive student behavior. 

... racial conflict among students. 

... student depression or mental health. 

... lack of respect for staff by students. 

... cutting classes or being truant. 

... gang-related activity. 

... weapons possession. 

... vandalism. 

... theft. 


This school provides resources ... 

... for staff to do their jobs well. 

... to work with special education (IEP) students. 


Source: Adapted with permission from California School Climate, Health, and Learning Survey [middle school staff survey], by WestEd, 2010. 
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Table 1. Overlap of school climate domains identified on the student and staff surveys of the California 
School Climate, Health, and Learning Survey 

Student survey domains 


Staff survey domains 



Support 
and safety 

Caring staff- Professional Student 

student Staff-peer development health and 

relationships relationships needs engagement 

Student 

delinquency 

Resource 

provision 

Safety and connectedness 

✓ 

✓ 



Caring relationships with adults 

✓ 



Meaningful participation 

✓ 




Substance use at school 



✓ 


Bullying and discrimination 



✓ 


Delinquency 



✓ 


Source: 2008/09 and 2009/10 Cal-SCHLS student and staff survey. 


The measure of student delinquency on the staff survey is more general than the related 
measure on the student survey. On the staff survey, student delinquency is assessed by 
13 questions that ask respondents to rate the extent to which substance use, harassment 
and bullying, physical violence, student depression, truancy, vandalism, and theft are a 
problem at the school. 

The analysis identified a distinct four-item student health and engagement domain on 
the staff survey, an indicator of student readiness to learn. Finally, the staff survey assess- 
es three domains not measured by the student survey: staff-peer relationships and two 
resource-related factors — a general factor for professional development needs (10 items) 
and a two-item measure of resource provision. 

How reliably do the survey items measure school climate at the respondent and school levels? 

Respondent-level reliability indicates how well a measure can be used to assess individu- 
al differences between students or teachers on that measure or differences between stu- 
dents and between teachers who score above and below some threshold. The higher the 
respondent-level reliability, the more likely the measure can be used to correctly distin- 
guish students who report low levels of safety and connectedness from students who report 
average levels of safety and connectedness, for example. Typically, a measure that includes 
more items yields greater respondent-level reliability. 

School-level reliability indicates how well a measure can be used to distinguish between 
schools in their average levels on the measure. The higher the school-level reliability, the 
more consistently the measure distinguishes schools with low scores from schools with 
high scores. School-level reliability rises with the number of surveys completed in a school, 
variation in average scores across schools, and respondent-level reliability. Information 
about school-level reliability can guide decisions about how many surveys to administer 
within schools. Such reliability information is also relevant for assessing the survey’s use- 
fulness for ranking and distinguishing schools. 

Student survey. To determine the respondent-level reliability of the measures derived 
from analysis of the student sample for each of the six domains, the study team calculated 
internal consistency reliability estimates using Cronbach’s alpha (a) (table 2). All reliability 
measures exceeded Nunnaly’s (1978) threshold of .70 for acceptable internal consistency, 
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Table 2. Student- and school-level reliability of student survey measures of the 
California School Climate, Health, and Learning Survey 



Student 

level 



School level 





r given size of student sample per school 11 


School climate domain 

Cron bach’s 
alpha 3 

226 

(mean) 

50 

100 

300 

400 

Intraclass 

correlation 

Safety and connectedness 

.80 

.95 

.81 

.89 

.96 

.97 

.078 

Caring relationships with adults 

.85 

.86 

.58 

.74 

.89 

.92 

.027 

Meaningful participation 

.68 

.89 

.63 

.77 

.91 

.93 

.033 

Substance use at school 

.88 

.81 

.49 

.66 

.85 

.89 

.019 

Bullying and discrimination 

.85 

.81 

.49 

.66 

.85 

.89 

.019 

Delinquency 

.83 

.89 

.63 

.77 

.91 

.93 

.033 


Note: The analytic sample consists of 214,465 students in 949 middle schools that administered the survey 
in 2008/09 or 2009/10. 

a. Recommended reliability coefficient of .70 or higher. 

b. Reliability predicted from Spearman-Brown formula; recommended value of .60 or higher. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


except the one for the meaningful participation domain (a = .68). The other measures 
demonstrated good internal consistency reliability, indicating that all survey items consis- 
tently measure a single underlying domain and can be used to consistently assess individu- 
al differences between students. 

The estimated school-level reliability coefficients (r) are above .80 for all of the measures 
when 226 students are surveyed at a school (the mean for the analytic sample; see table 2). 
This result suggests that, with a sample of approximately 200 students in a school, all six 
measures are reliable when aggregated to the school level. With a sample of 100 students in 
a school, the school-level reliability for all measures is reduced but is still above .60. With 
school sample sizes of 50 students, the school-level reliability for caring relationships with 
adults, substance use at school, and bullying and discrimination falls below the Fastenau 
et al. (1996) criterion of .60. Thus, a school-specific sample of at least 100 students is neces- 
sary to ensure adequate school-level reliability. 

The intraclass correlation captures the extent to which students’ scores are similar in 
the same school or, stated differently, the proportion of variation in each measure that 
is located between rather than within schools. For example, an intraclass correlation of 
.30 indicates that 30 percent of the total variation is between schools and 70 percent is 
within schools. The correlations for the school climate scales in the student survey range 
from .019 for bullying and discrimination and substance use at school to .078 for safety and 
connectedness (see table 2). These values indicate that most of the variation in student 
perceptions of school climate is within rather than between schools. In addition, students’ 
scores are more similar within the same school for safety and connectedness than for bul- 
lying and discrimination and substance use at school. Falling between those two intraclass 
correlations are the within-school consistency of scores on caring relationships with adults 
(.027), meaningful participation (.033), and delinquency (.033). 

Staff survey. All seven of the staff survey measures for teachers demonstrated internal 
consistency reliabilities exceeding Nunnaly’s (1978) .70 threshold for acceptable internal 


All reliability 
measures except 
the one for the 
meaningful 
participation 
domain 
demonstrated 
good internal 
consistency 
reliability, 
indicating that 
all survey items 
consistently 
measure a single 
underlying domain 
and can be used 
to consistently 
assess individual 
differences 
between students 
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consistency (table 3). The internal consistency estimates range from .82 (resource provi- 
sion) to .96 (support and safety). According to these results, teacher responses to the staff 
survey consistently measure each school climate domain. 

For a sample size of 19 teachers at a school (the mean of the analytic sample), all the 
school-level reliability coefficients exceed .70 except the coefficient for professional devel- 
opment needs (r = .50) (see table 3). A sample size of 30 or more is needed to achieve ade- 
quate reliability for the professional development needs item (r = .61). Thus, a sample size 
of 10 or more teachers is needed to consistently distinguish schools with low levels from 
those with high levels for all the staff-reported school climate measures except professional 
development needs. 


The intraclass correlations are higher for the staff measures than for the student measures 
(except for professional development needs), indicating that within the same school, teach- 
ers’ scores are more similar than are student scores. Across the teacher measure, the intra- 
class correlations are highest for student health and engagement (.352), followed by support 
and safety (.263) and student delinquency (.152). Professional development needs had the 
lowest intraclass correlation (.050), indicating that teachers within a school exhibit more 
dissimilar scores on this measure than on the other measures. The professional develop- 
ment survey questions ask teachers to report on their own needs for professional develop- 
ment rather than on their perceptions about the school, which may explain why responses 
were more dissimilar on this measure than on the other staff survey measures. 

How stable are the school-level climate measures? 


The study examined the stability of school-level climate scores over one- and two-year 
intervals to identify the persistence of school climate features and to estimate a lower 


Table 3. Teacher- and school-level reliability of staff survey measures of the 
California School Climate, Health, and Learning Survey 

School climate domain 

Teacher 

level 



School level 



Cronbach’s 

r given teacher sample size per school b 

Intraclass 

alpha 3 

19 (mean) 

5 

10 

30 

40 

correlation 

Support and safety 

.96 

.87 

.64 

.78 

.91 

.93 

.263 

Caring staff-student 

.95 

.74 

.43 

.60 

.82 

.86 

.129 

relationships 








Staff-peer relationships 

.88 

.75 

.44 

.61 

.83 

.86 

.136 

Professional 

.89 

.50 

.21 

.34 

.61 

.68 

.050 

development needs 








Student health and 

.84 

.88 

.67 

.80 

.92 

.94 

.352 

engagement 








Student delinquency 

.91 

.91 

.73 

.84 

.94 

.96 

.152 

Resource provision 

.82 

.77 

.47 

.64 

.84 

.88 

.129 


Note: The analytic sample consists of 14,347 teachers in 764 middle schools that administered the survey in 
2008/09 or 2009/10. 

a. Recommended reliability coefficient of at least .70. 

b. r = reliability predicted from Spearman-Brown formula; recommended value of at least .60. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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bound of test-retest reliability. These stability estimates, based on a large sample of middle 
schools, can be a yardstick for interpreting changes in schools that are working to improve 
the school climate. 


Student survey. Average correlations between school-level school climate measures over 
one- and two-year intervals were weighted by the number of schools that administered 
the student survey at each interval between 2004/05 and 2010/11 (see tables D9 and D10 
in appendix D for estimated correlations for each interval). The average correlation of 
the student-reported measures over one-year intervals ranged from .36 (substance use at 
school) to .80 (safety and connectedness), with an average of .55 across all the student 
survey measures (table 4). This compares with an average correlation of .64 across grade 7 
school climate measures in Brand et alls (2003) analysis of school-level stability. 

Stability over one-year intervals was greater between the 2009/10 and 2010/11 adminis- 
trations of the survey than between earlier administrations (see table D9 in appendix D). 
Between the 2009/10 and 2010/11 administrations, the stability correlation for all measures 
was at least .58 (for substance use) and above .70 for four other measures (it was .62 for 
bullying and discrimination, the sixth measure). 

The correlations for the student survey results were weaker for the two-year intervals than 
for the one-year intervals (see table 4), ranging from .34 to .68. The difference in stabil- 
ity was largest for safety and connectedness (.12). For the other measures the difference 
between one- and two-year correlations ranged from .02 to .08. The average two-year 


Table 4. Stability of school climate measures for the California School Climate, 

Health, and Learning Survey at one- 

and two-year intervals (correlations) 

Measure 

One-year interval 

Two-year interval 

Student survey 

Safety and connectedness 

.80 

.68 

Caring relationships with adults 

.52 

.45 

Meaningful participation 

.54 

.51 

Substance use at school 

.36 

.34 

Bullying and discrimination 

.49 

.41 

Delinquency 

.61 

.56 

Number of survey intervals 

255 

1,941 

Staff survey 

Support and safety 

.62 

.42 

Caring staff-student relationships 

.69 

.45 

Staff-peer relationships 

.78 

.38 

Professional development needs 

-.07 

.43 

Student health and engagement 

.72 

.66 

Student delinquency 

.69 

.64 

Resource provision 

.56 

.42 

Number of survey intervals 

42 

119 


Note: The student survey analytic sample consists of 129 schools that administered the student survey in 
consecutive years and 959 schools that administered it at two-year intervals. The staff survey analytic sample 
consists of 34 schools that administered the staff survey in consecutive years and 119 schools that adminis- 
tered it at two-year intervals. 

Source: 2004/05 to 2010/11 CAL-SCHLS student survey and 2008/09 to 2010/11 staff survey. 
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correlation of .49 was similar to the correlation of .445 estimated by Brand et al. (2003). For 
the student survey, safety and connectedness was the most stable of the schooUevel mea- 
sures, followed by delinquency, meaningful participation, caring relationships with adults, 
bullying and discrimination, and substance use at school. 

Staff survey. The average correlation of the teacher measures over one-year intervals 
ranged from -.07 for professional development needs to .78 for staff-peer relationships (see 
table 4). With the exception of professional development needs, an outlier, all variables 
had correlation averages of .56 or higher. An average correlation of close to zero for pro- 
fessional development needs suggests that it is not a reliable measure of a stable school 
characteristic. The two-year correlations were lower than the one-year correlations for six 
of the seven teacher measures. They ranged from .38 for staff-peer relationships to .66 for 
student health and engagement. Student health and engagement and student delinquency 
both had one- and two-year average correlations above .60. Comparisons between one- 
and two-year correlations should be interpreted cautiously because only 34 schools admin- 
istered the staff survey in consecutive years. 

Are the school climate measures related to student achievement and discipline infractions? 

The predictive validity of the school climate measures was analyzed by examining their 
correlations with school-level measures of academic achievement and suspensions. Cor- 
relations of .30 or higher are assumed to provide evidence of predictive validity (Biddle, 
2005). 

Student survey. High scores on safety and connectedness, caring relationships with adults, 
and meaningful participation were expected to correlate positively with English language 
arts and math test scores and negatively with suspension rates. The opposite was expected 
for high scores on substance use, bullying and discrimination, and delinquency. 

The correlations were in the expected direction for all six measures (table 5). All student- 
reported measures except bullying and discrimination had a correlation of .30 or higher 
with each achievement test score variable, with safety and connectedness and delinquency 
showing the strongest correlations. All the measures except substance use and bullying 
and discrimination had a correlation of .30 or higher with suspensions per 100 students. 
The correlations of the student survey measures were consistently lower for suspensions 
than for academic achievement. 

Predictive validity is supported for all of the student-reported measures except bullying 
and discrimination; the correlation of bullying and discrimination with other measures 
of school functioning never exceeded .30. The correlations of school climate scores with 
grade 7 and 8 English language arts scores were very similar, even though the school 
climate scores are based solely on the grade 7 student reports. 

Staff survey. High scores on support and safety, caring staff-student relationships, 
staff-peer relationships, student health and engagement, and resource provision were 
expected to correlate positively with English language arts and math test scores and 
negatively with suspensions per 100 students. The opposite was expected for profession- 
al development needs and student delinquency. The correlations between school-level 
staff survey measures and achievement and suspensions were in the expected direction 


Predictive validity 
is supported for 
all of the student- 
reported measures 
except bullying and 
discrimination 
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Table 5. School-level correlations of school climate measures on the California 
School Climate, Health, and Learning Survey with student achievement and 
discipline infractions 


English language arts scores 

Math scores 


Measure 

Grade 7 

Grade 8 

Grade 7 

Suspensions 

Student survey 

Safety and connectedness 

.66 

.65 

.60 

-.46 

Caring relationships with adults 

.43 

.42 

.41 

-.30 

Meaningful participation 

.50 

.50 

.47 

-.33 

Substance use 

-.46 

-.44 

-.39 

.29 

Bullying and discrimination 

-.18 

-.20 

-.23 

.13 

Delinquency 

-.60 

-.61 

-.53 

.35 

Staff survey 

Support and safety 

.45 

.47 

.43 

-.22 

Caring staff-student relationships 

.40 

.41 

.35 

-.18 

Staff-peer relationships 

.27 

.28 

.23 

-.14 

Professional development needs 

-.32 

-.33 

-.29 

.16 

Student health and engagement 

.77 

.77 

.68 

-.43 

Student delinquency 

-.67 

-.69 

-.60 

.45 

Resource provision 

.25 

.26 

.30 

-.15 


Note: The student survey analytic sample consists of in 949 middle schools. The staff survey analytic sample 
consists of 764 middle schools that administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey and staff survey. 
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within schools 


for all seven measures (see table 5). Across teacher-reported measures, student health 
and engagement and student delinquency had the strongest average correlations, fol- 
lowed by support and safety and caring staff-student relationships. Staff-peer relation- 
ships, resource provision, and professional development needs had the weakest average 
correlations with test scores and suspensions. However, in most cases these correlations 
still approached .30. The correlations were consistently lower for suspensions than for 
academic achievement. 


Study implications 


For reliable school-level measurement of school climate, the study findings suggest that 
schools should sample at least 100 grade 7 students 2 and 10 teachers in the school. 3 The 
respondent- and school-level reliability results suggest that both the student- and teacher- 
reported measures can be used to distinguish differences between schools and between 
respondents within schools. The poor reliability (r = .50) of the measure for professional 
development needs suggests that this domain should not be included in measuring school 
climate. 

As with student standardized test scores and other school-based measures, most of the 
variation in student and teacher perceptions of school climate was found within rather 
than between schools. Thus, the measures may be particularly useful for identifying differ- 
ences in student perceptions of school climate across different groups (for example, racial/ 
ethnic groups) within schools. Differences between schools with the lowest and highest 
scores may not be substantively meaningful because there is so little variation between 
schools for some school climate domains. 
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Study limitations 


The CahSCHLS student survey was designed primarily as an epidemiological surveillance 
survey for school districts to track school climate factors, student health risks, and student 
problem behaviors that research has identified as important barriers to learning (Hanson 
& Kim, 2007). It was designed to measure a broad array of factors while being short enough 
for widespread administration in schools. That intentional brevity may mean that there 
are too few items to use the survey to adequately assess the underlying factors of some 
school climate domains. Another limitation of the data for assessing school climate in 
middle school is that only grade 7 students were surveyed. To the extent that students in 
grades 6 and 8 have different perceptions of the school climate, the student survey data 
provide only a partial picture of the school environment as perceived by middle school 
students. However, schooHevel scores would be expected to remain fairly stable across dif' 
ferent cohorts of grade 7 students if the school climate scales capture enduring features of 
the school environment. 

The structures of the student and teacher samples differ in important ways that have 
implications for comparing the student and teacher results. First, student survey data were 
collected from grade 7 students exclusively, while staff survey data were collected from 
teachers in all middle school grades. This may reduce the student-based measures’ com- 
parability with the teacher measures. Also, because the survey targeted grade 7 students 
exclusively, the same students are unlikely to participate in the grade 7 survey in subse- 
quent years. Most teachers, however, remain at the same school in successive years and may 
thus participate in the survey year after year. This could inflate the stability estimates of 
the teacher measures relative to that of the student measures. If there are stable differences 
in individual perceptions of school climate, then surveying the same group of respondents 
in successive years (as in the case of teachers) will yield more consistent responses than 
asking different groups of respondents (as with the students). Another reason to interpret 
the stability analyses cautiously is that only a small number of schools administered the 
staff survey in consecutive years, reducing the results’ precision and generalizability. 

A further limitation of the data is that Cal-SCHLS student and teacher response rates vary 
substantially across schools. Sensitivity analyses conducted to determine whether results 
differed between the total sample of schools and the sample of schools with high and low 
response rates found that the response rate did not affect the school climate domains iden- 
tified or the respondent-level reliability estimates of the measures. And with one exception 
(substance use at school), school-level reliability was similar in the two samples. However, 
the student-reported measures were more stable and more strongly associated with student 
achievement and suspension rates when response rates were high than when they were 
low. Such differences in stability and predictive validity were not apparent for the teach- 
er-reported measures. These results suggest that the stability and validity of the student 
survey school climate measures are enhanced when response rates are high. 
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Appendix A. The link between school climate and student learning 


School climate encompasses the characteristics of a school’s environment that influence 
students’ academic and social development (Brand et al., 2008). A more positive school 
climate is associated with more positive student outcomes. Students in schools with safe, 
caring, and participatory climates show higher academic performance (Bowen & Bowen, 
1999; Hanson et al., 2004; Voight et al, 2013) and stronger socioemotional competence 
(Anderson, 1982; Blum et al., 2002; Eccles et al., 1993). Poor school climates are associ- 
ated with lower student engagement and attendance, more frequent disruptive classroom 
behavior, lower academic aspirations, and diminished performance (Benninga et al., 2003; 
Dejung & Duckworth, 1986; Rumberger, 1987; Wu et al, 1982). 

School climate domains 

Studies have identified multiple domains of school climate (Bear, Gaskins, Blank, & Chen, 
2011; Brand et al, 2008; Cohen, McCabe, Michelli, & Pickeral, 2009): 

• The quality of relationships among and between students and staff. 

• The degree of connectedness that students feel to the school community. 

• Expectations that the school explicitly and implicitly sets for student learning and 
behavior. 

• Opportunities for students to participate in meaningful and personally relevant 
activities. 

• How safe students and staff feel at school, including how safe they feel from bully- 
ing, harassment, and other physical and emotional forms of victimization. 

• The quality of the school facility and its physical environment. 

To measure school climate, schools and districts often administer surveys to students and 
staff. Some school climate survey questions ask respondents about their level of agreement 
with specific statements about the school’s social and physical environment (Brand, 2003). 
Other questions are based on group-level aggregation of respondent reports of their own 
behaviors and attitudes (Liidtke et al, 2008). 

The federal Safe and Supportive Schools model 

In 2009 the U.S. Department of Education’s Office of Safe and Drug Free Schools drafted 
a model of school climate that includes three main categories: engagement, safety, and 
environment (figure Al) (Jennings, 2010; O’Malley, Renshaw, Ritchey, & Furlong, 2011). 

Engagement. Engagement includes relationships, respect for diversity, and school partic- 
ipation. High-quality relationships between and among teachers and students are critical 
for developing a school environment conducive to student attachment and learning. For 
example, clear and positive teacher expectations structure and guide behavior and provide 
the challenge necessary to press students to succeed (Benard, 2004). Students benefit 
most when high expectations are coupled with high levels of teacher warmth and support 
(Bryk et al., 2010; Shouse, 1996). Caring teacher-student relationships are positively 
linked to student engagement in school, academic performance, and personal well-being 
(Benard, 2004; Danielsen, Samdal, Hetland, & Wold, 2009; Malecki & Demaray, 2003). 
Students are more likely to internalize the values of those to whom they feel a strong 
sense of attachment (Ryan & Deci, 2000). Student attachment to teachers is fostered when 
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Figure Al. Federal Safe and Supportive Schools model 



Source: Safe and Supportive Schools Technical Assistance Center 2012. 


teachers exhibit high levels of warm, supportive behavior. Consequently, students whose 
teachers demonstrate caring behavior are more likely to internalize the value of succeeding 
in school (Danielsen, Wiium, Wilhelmsen, & Wold, 2010). 

Respect for diversity — defined as the acceptance of subgroups in the school and the 
community — is thought to promote a positive school climate by enhancing school safety 
and the quality of student relationships (Green, Adams, & Turner, 1988). Promoting the 
inclusiveness of all student groups may diminish or eliminate teasing and bullying behav- 
ior directed at students who are perceived to be different. Respect for diversity is likely an 
essential step to addressing racial/ethnic disparities in students’ school engagement (Cohen 
et al., 2009). 

School participation refers to staff, student, and parent participation in school activities 
and governance, as well as to broader community involvement in school activities (Cohen 
et al., 2009; Libbey, 2004). Learning environments that support autonomy — that provide 
opportunities for students to engage in relevant and interesting activities that encour- 
age responsibility and elicit contribution — are believed to foster motivation and promote 
identification with the goals of learning (Black & Deci, 2000). When students are given 
responsibility and some control over the learning process, they are more likely to become 
invested in academic and other types of success (Newman, 1992). Moreover, participating 
meaningfully in the classroom allows students to contribute actively and positively to the 
classroom climate and likely enhances their bonds with the school (McLaughlin, 2000; 
National Research Council, 2003). 

Safety. School safety includes emotional and physical safety, as well as low rates of sub- 
stance use. Numerous studies demonstrate that bullying and violence in schools harm 
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students (Beauvais, Chavez, Oetting, Deffenbacher, & Cornell, 1996; Bowen & Bowen, 
1999; Bowen, Richman, Brester, & Bowen, 1998; Eccles, Lord, & Midgley, 1991; Ellickson, 
Saner, & McGuigan, 1997; Furlong, Chung, Bates, & Morrison, 1995; Gronna & Chin- 
Chance, 1999; Herrenkohl et al, 2000; Jenkins & Bell, 1994; National Center for Educa- 
tional Statistics, 1995). 

Unsafe school environments can affect student performance and learning in several ways. 
First, exposure to violence and abuse can increase emotional and psychological distress, 
making it harder for students to concentrate and attend to academic matters. Second, 
the distress associated with exposure to crime, violence, bullying, and teasing may reduce 
instruction time by causing students to stay home from school or cut classes (Harris & 
Associates, 1995). Third, crime, violence, and social disorganization at school may affect 
academic performance by influencing teaching and learning processes in the classroom. 
For example, having disruptive and aggressive students in the classroom diverts teachers’ 
attention, reducing instruction time for all students (Lochman, Lampron, Gemmer, & 
Harris, 1987, as cited in Bowen & Bowen, 1999). 

Environment. Environment consists of wellness and the physical, academic, and disci- 
plinary environments. Wellness refers to the availability of physical and mental health 
supports in school for students with health-related problems that make learning difficult. 

The physical environment is the physical space of the school, including facility quality and 
upkeep, ambient noise, and classroom temperature. These aspects of a school’s physical 
environment are associated with academic performance and other school climate domains 
(Evans, 2006; Plank, Bradshaw, & Young, 2009). Middle school students who attend 
schools that are in disrepair report less positive relationships with teachers and other stu- 
dents, lower perceptions of safety, and higher perceptions of social disorder (for example, 
fighting, verbal abuse, substance use) at school (Plank et al., 2009). 

The academic environment consists of classroom instructional and behavioral features, 
such as teaching and enforcing classroom rules and routines, adapting instructional strate- 
gies to increase opportunities for academic success, and teaching and reinforcing appropri- 
ate behavior and social skills (Epstein, Atkins, Cullinan, Kutash, & Weaver, 2008). 

The classroom behavioral and academic environment is inextricably linked to student 
learning. Finally, the disciplinary environment refers to the communication to students 
of school policies on discipline and behavior, and the adequacy and fairness of policy 
enforcement. 
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Appendix B. School climate items on the California School 
Climate, Health, and Learning Survey student and staff surveys 


The student survey of the California School Climate, Health, and Learning Survey 
(CahSCHLS) includes 115 items in its core module that assess students’ perceptions of 
their school environment and their personal health and risk behaviors. Only the 40 core 
module items that are relevant to students’ in-school experiences are included in this study 
(table Bl). 

The staff survey of the CahSCHLS consists of three sections: the first for all staff and the 
other two for specialized staff. The study used only the first section, which encompasses 
69 items designed to assess five school climate areas: safe learning environments, norms 
and standards that encourage student learning, quality of relationships between staff and 
between staff and students, student behaviors that facilitate learning, and services and 
programs that address nonacademic barriers to learning (table B2). 


Table Bl. School climate items on the student survey of the California School 
Climate, Health, and Learning Survey that are relevant to students’ in-school 
experiences 


Question 

number 

Description 


11 

1 feel close to people at this school. 


12 

1 am happy to be at this school. 


13 

1 feel like 1 am a part of this school. 


14 

The teachers at this school treat students fairly. 


15 

1 feel safe in my school. 


At this school there is an adult who. . . 

16 

... really cares about me. 


17 

... tells me when 1 do a good job. 


18 

... notices when 1 am not there. 


19 

... always wants me to do my best. 


20 

... listens to me when 1 have something to say. 


21 

... believes that 1 will be a success. 


At school... 

22 

... 1 do interesting activities. 


23 

... 1 help decide things like activities or rules. 


24 

... Ido things that make a difference. 


Substance use at school 

55 

During your life, how many times have you been drunk on alcohol or 
property? 

“high” on drugs on school 

During the past 30 days, on how many days on school property have you... 

72 

... smoked cigarettes? 


73 

... had at least one drink of alcohol? 


74 

... smoked marijuana? 


75 

... used any other illegal drug or pill to get “high?’’ 


During the past 12 months, how many times on school property have you... 

100 

... been pushed, shoved, or slapped? 


101 

... been afraid of being beaten up? 


(continued) 
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Table Bl. School climate items on the student survey of the California School 
Climate, Health, and Learning Survey that are relevant to students’ in-school 
experiences (continued) 


Question 

number 

Description 

102 

... been in a physical fight? 

103 

... had rumors or lies spread about you? 

104 

... had sexual jokes, comments, or gestures made to you? 

105 

... been made fun of because of your looks or the way you talk? 

106 

... had your property stolen or deliberately damaged, such as your car, clothing, or books? 

107 

... been offered, sold, or given an illegal drug? 

108 

... damaged school property on purpose? 

109 

... carried a gun? 

110 

... carried any other weapon (such as a knife or club)? 

111 

... been threatened or injured by a weapon (such as a knife or club)? 

112 

... seen someone carrying a gun, knife, club, or other weapon? 

During the past 12 months, how many times on school property were you harassed or bullied for any of the 
following reasons? [You were bullied if repeatedly shoved, hit, threatened, called mean names, teased in a way 
you didn’t like, or had other unpleasant things done to you. It is not bullying when two students of about the 
same strength quarrel or fight.] 

113 

Your race, ethnicity, or national origin 

114 

Your religion 

115 

Your gender (being male or female) 

116 

Because you are gay or lesbian or someone thought you were 

117 

A physical or mental disability 

118 

Any other reason 

119 

How safe do you feel at school? 

120 

In the past 12 months, 1 have been cyber-bullied 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


Table B2. School climate items from the first section of the staff survey of the 
California School Climate, Health, and Learning Survey 

Question 

number 

Description 

This school... 

6 

... is a supportive and inviting place for students to learn. 

7 

... sets high standards for academic performance for all students. 

8 

... promotes academic success for all students. 

9 

... emphasizes helping students academically when they need it. 

10 

... provides adequate counseling and support services for students. 

11 

... emphasizes teaching lessons in ways relevant to students. 

12 

... is a supportive and inviting place for staff to work. 

13 

... promotes trust and collegiality among staff. 

14 

... provides the materials, resources, and training (professional development) needed to do your 
job effectively. 

15 

... provides the materials, resources, and training (professional development) needed to work 
with special education (IEP) students. 

(continued) 
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Table B2. School climate items from the first section of the staff survey of the 
California School Climate, Health, and Learning Survey staff survey (continued) 

Question 

number 

Description 

16 

... encourages opportunities for students to decide things like class activities or rules. 

17 

... gives all students equal opportunity to participate in classroom discussions or activities. 

18 

... gives all students equal opportunity to participate in numerous extracurricular and 
enrichment activities. 

19 

... gives students opportunities to “make a difference” by helping other people, the school, or 
the community (e.g., service learning). 

20 

... encourages students to enroll in rigorous courses (such as honors and AP), regardless of 
their race, ethnicity, or nationality. 

21 

... emphasizes using instructional materials that reflect the culture or ethnicity of its students. 

22 

... has staff examine their own cultural biases through professional development and other 
processes. 

23 

... considers closing the racial/ethnic achievement gap a high priority. 

24 

... fosters appreciation for student diversity and respect for each other. 

25 

... emphasizes showing respect for all students' cultural beliefs and practices. 

26 

... clearly communicates to students the consequences of breaking school rules. 

27 

... handles discipline problems fairly. 

28 

... effectively handles student discipline and behavioral problems. 

29 

... is a safe place for students. 

30 

... is a safe place for staff. 

31 

... is welcoming to and facilitates parent involvement. 

32 

... has clean and well-maintained facilities and property. 

How many adults in this school... 

33 

... really care about every student? 

34 

... acknowledge and pay attention to students? 

35 

... want every student to do their best? 

36 

... listen to what students have to say? 

37 

... believe that every student can be a success? 

38 

... treat all students fairly? 

39 

... treat every student with respect? 

40 

... have close professional relationships with one another? 

41 

... support and treat each other with respect? 

42 

... feel a responsibility to improve this school? 

Do you feel that you need more professional development, training, mentorship, or other support to do your 
job in any of the following areas? 

43 

Meeting academic standards 

44 

Evidence-based methods of instruction 

45 

Positive behavioral supports/classroom management 

46 

Working with diverse racial, ethnic, or cultural groups 

47 

Culturally relevant pedagogy for the school’s student population 

48 

Serving English language learners 

49 

Closing the achievement gap 

50 

Serving special education (IEP) students 

51 

Meeting the social, emotional, and developmental needs of youth (e.g., resilience promotion) 

52 

Creating a positive school environment 


(continued) 
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Table B2. School climate items from the first section of the staff survey of the 
California School Climate, Health, and Learning Survey staff survey (continued) 

Question 

number 

Description 

Based on your experience, how many students at this school... 

53 

... are healthy and physically fit? 

54 

... arrive at school alert and rested? 

55 

... are motivated to learn? 

56 

... are well-behaved? 

How much of a problem at this school is... 

57 

... student alcohol and drug use? 

58 

... student tobacco use? 

59 

... harassment or bullying among students? 

60 

... physical fighting between students? 

61 

... disruptive student behavior? 

62 

... racial/ethnic conflict among students? 

63 

... student depression or other mental health problems? 

64 

... lack of respect for staff by students? 

65 

... cutting classes or being truant? 

66 

... gang-related activity? 

67 

... weapons possession? 

68 

... vandalism? 

69 

... theft? 


Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Appendix C. Methodology 


The study relied on existing data from California public schools’ administration of the 
California School Climate, Health, and Learning Survey (CahSCHLS) to grade 7 student 
and middle school staff. Separate analyses were conducted for the student survey and the 
teacher responses to the staff survey. 

Data 

Since 1997 WestEd’s Health and Human Development Program, on contract to the Cal- 
ifornia Department of Education, has provided school districts administering the surveys 
with technical assistance, data scanning, and a report on the district-level data collected 
in each module. These data, collected anonymously, are housed in WestEd’s Health and 
Human Development Program. The California Department of Education approved the use 
of survey data to conduct this research. School data for academic performance and suspen- 
sions are publicly available on the California Department of Education’s website. 

School climate data used in this study are pooled from administration of the CahSCHLS 
student survey instrument in California middle schools 4 from 2004/05 through 2010/11 
and from the staff survey instrument from 2008/09 through 2010/11. The CahSCHLS staff 
survey is administered to all types of school staff (teachers, administrators, classified staff), 
but only reports from teachers were included in this study. 

The study addressed four research questions: 

• What school climate domains in middle school do the survey items measure? 

• How reliably do the surveys items measure school climate at the respondent and 
school levels? 

• How stable are the schoohlevel climate measures? 

• Are the school climate measures related to student achievement and discipline 
infractions? 

The analyses that address research questions 1, 2, and 4 are based on data collected from 
214,465 grade 7 students in 997 middle schools that administered the student survey and 
14,806 teachers in 794 middle schools that administered the staff survey in the 2008/09 or 
2009/10 academic years — the most recent period in which the participating schools and 
districts were representative of the schools and districts in the state. 

These students, teachers, and schools represent a fraction of those in the larger dataset that 
includes survey results from 2004/05 to 2010/11. During 2008/09 and 2009/10, the California 
Department of Education required all schools and districts receiving Title IV funding under 
the No Child Left Behind Act to administer the student and staff surveys at least once every 
two years. Middle schools were required to administer the student survey to grade 7 students 
and the staff survey to all school staff. Also during this period, 48 middle schools adminis- 
tered the student survey in both 2008/09 and 2009/10, and 30 middle schools administered 
the staff survey in both years. For these schools only data from 2009/10 were included in the 
analyses for research questions 1, 2, and 4. The 2008/09 data from schools that administered 
the survey more than once during the two-year period were excluded to ensure that the 
school-level observations in the dataset were independent (see below) and that the sample 
was representative of schools that were required to administer the surveys. 
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Analyses for research questions 1 and 2 were based on respondent-level data (either student 
or teacher); analyses for research questions 2, 3, and 4 were based on data aggregated to the 
school level. To examine research question 3 on stability, the study team used Cal-SCHLS 
grade 7 student survey data from 962 middle schools from 2004/05 through 2010/11 and 
teacher data from 145 middle schools for each school year from 2008/09 through 2010/11. 
Only three years of teacher data were used in the stability analyses because many new 
items were added to the CahSCHLS staff survey after 2007/08. 

School performance data from the California Department of Education’s Standardized 
Testing and Reporting program were used to examine the relationship of school climate 
to academic performance (see box 1 in the main report). Academic performance is based 
on test score data aggregated at the school level. School-level suspension and expulsion 
data came from the California Department of Education’s Uniform Management Infor- 
mation Reporting System database. Suspension and expulsion counts were converted into 
rates per 100 students. After discovering that the variation across schools was low — nearly 
twO'thirds of schools had fewer than five total expulsions — the analyses examining the 
relationship of school climate to expulsion rates were dropped. 

Analyses 

For each research question separate analyses were conducted for the student sample and 
the teacher sample. The analytic approaches discussed below were applied to both the 
student and teacher datasets. 

What school climate domains in middle school do the survey items measure ? To address 
research question 1, the study team estimated a series of exploratory factor analysis (EFA) and 
confirmatory factor analysis (CFA) models. EFA models were estimated using oblique rotation 
to determine roughly the number of latent factors underlying the data and their measurement 
structure, as well as how the survey items mapped onto the underlying dimensions identified. 
A random split-half sample of schools from the full sample was used for the EFA, and the 
other half was used as a sample for CFA, described below. The results of the EFA models were 
used as a starting point for a series of nested CFA models with the validation sample. Factor- 
loading patterns, measures of model fit, and correlations among the latent factors or tests of 
discriminant validity were used to make decisions about the final models (table Cl). 

Specifically, the optimal number of factors in the EFA was determined by first estimat- 
ing a series of models, each specifying a different numbers of factors, and then comparing 


Table Cl. Steps in selecting a measurement model 


Step 1 

Estimate 10 exploratory factor analysis models with 1-10 factors using first split-half sample of schools. 

Step 2 

Choose model that has RMSEA < .06, SRMR < .08, CFI > .95, and TLI > .95. 

Step 3 

Estimate confirmatory factor analysis model based on step 2 using second split-half sample of schools. 

Step 4 

Modify confirmatory factor analysis based on factor-loading patterns observed in steps 2 and 3. 

Step 5 

Choose model that has RMSEA <.06, lowest WRMR, CFI > .95, and TLI > .95. 

Step 6 

Combine factors that fail Fornell and Larcker's (1981) discriminant validity test. 


RMSEA is root mean square error of approximation. SRMR is standardized root mean square residual. CFI is 
comparative fit index. TLI is Tucker-Lewis index. WRMR is weighted root mean square residual. 

Source: Authors' construction. 
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indices of model fit to see how well each model represents the actual data (step 1, table Cl). 
Several common fit indices were used to help with this determination, including the root 
mean square error of approximation (RMSEA), the standardized root mean square residual 
(SRMR), the comparative fit index (CFI), and the Tucker-Lewis index (TLI). A smaller 
value of RMSEA and SRMR and larger value of CFI and TLI indicate better model fit. 
The rule of thumb for a “good fit’ model is a value of .06 or lower for the RMSEA, 5 .08 
or lower for the SRMR, and .95 or higher for the CFI and TLI (Hu & Bentler, 1999). 
The decisionmaking criterion for choosing the number of factors in all EFAs was to select 
the model with the fewest number of factors that still met the cutoff for good fit for the 
RMSEA, SRMR, CFI, and TLI indicators (step 2, table Cl). This approach privileges 
simpler factor structures while still ensuring sound psychometric qualities. 

The EFA solution was used as a starting point for a series of nested CFA models using data 
from a sample that included only the schools not selected for the EFA analytic sample (steps 
3-4, table Cl). The CFA models were estimated not to confirm the estimated EFA model 
but to identify a conceptually grounded, best-fitting, sensible model (Bollen, 1989). Similar 
to other scale development work, the CFA was used in an exploratory fashion to ascertain 
the best use of the survey items for identifying conceptually distinct school climate domains 
(Brand et al., 2003; Cabrera-Nguyen, 2010; Worthington & Whittaker, 2006). Because the 
analytic samples are so large, modifying the CFA models sequentially using data from one 
sample is unlikely to be affected by chance sample characteristics (Hutchinson, 1998). 

The CFA models used the same fit indices and cutoff criteria as for the EFAs to assess 
model fit, with the exception that the SRMR was replaced by the weighted root mean 
square residual (WRMR). 6 A smaller WRMR indicates better fit. The conventional cutoff 
for good fit is 1.0 or lower, although the WRMR is often inflated in analyses with such 
large sample sizes (Yu, 2002). Models that had better estimated fit across the five indices 
were prioritized (step 5, table Cl). 

After selecting a preferred CFA model (step 5), the study team applied Fornell and Larcker’s 
(1981) test for assessing discriminant validity of factors (the extent to which latent variables 
adequately discriminate from other latent variables). This involves comparing the average 
variance explained by the latent factor on observed indicators with the shared variance 
explained by latent factors. 7 If the shared variance explained with any other factor is larger 
than the average variance explained by a latent factor, then discriminant validity is not sup- 
ported and, for this study, the two factors were combined into a single factor (step 6, table Cl). 

To derive estimates for the EFA and CFA models, the study employed Muthen and Muthen’s 
(2012) Mplus 7 statistical modeling program. Because items on the Cal-SCHLS are ordinal 
(that is, measured using Likert-type response options), Muthen’s (1984) approach to EFA 
and CFA with ordinal indicators was applied, where each observed dichotomous or ordinal 
item (y) is linked to a latent continuous item (y*) in a nonlinear way through a model of 
thresholds. The relationships between an observed ordinal item y with c categories to y* 
can be expressed as: 


y = c, if r c <y* < t +1 

for c = 0, 1, 2, ..., c-1. The rs represent threshold parameters. Muthen’s (1984) approach 
models the relationships among these more fundamental latent y* variables. 
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The EFAs and CFAs accounted for the nesting of students within schools using clustered 
errors. No covariates for year of survey administration were included in the CFA models 
because survey year is unlikely to be related to the measurement parameter estimates and, 
with mandatory biennial survey administration, schools that administered the survey over 
a two-year period represent all the schools with Cal-SCHLS data in the state. 

How reliably do the survey items measure school climate at the respondent and school 
levels ? Respondent-level internal consistency of reliability of each measure was estimated 
by calculating Cronbach’s alpha, using Nunnaly’s (1978) criterion of .70 as the benchmark 
for acceptable internal consistency reliability. Cronbach’s alpha assesses the extent to 
which each survey item used in a scale produces similar scores. 

The proportion of the total variance of each measure that is located between schools was 
first estimated to determine school-level reliability (the intraclass correlation), and the 
intraclass correlation was incorporated into the Spearman-Brown prediction formula for 
each scale (Raudenbush & Bryk, 2001; Winer et al., 1991). The reliability of the average 
school-level score as an estimator of the true school average was assessed using: 


1 = 
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where X is an estimate of the proportion of total variance that is true school variance, as 
opposed to variance due to measurement or sampling error, t qo is the between-group vari- 
ance, cr 2 is the within-group variability, and n. is the within-school sample size in a hypo- 
thetical school j (Raudenbush & Bryk, 2001). Fastenau et alls (1996) criterion of .60 was 
used as the threshold for acceptable school-level reliability. A range of reliability values 
corresponding to different school sample sizes is included in the results. 

How stable are the schooldevel climate measures ? To examine the stability of school 
aggregate measures over time, the scores for individual school climate measures were 
aggregated to the school level, and correlations were estimated across one- and two-year 
intervals of survey administration. Average correlations for all one- and two-year intervals 
were weighted by the number of survey administrations over the interval period. These 
estimates are presented in table 5 in main report. Tables D9-D11 in appendix D show 
stability results for each year of survey administration. Student data collected from schools 
over a seven-year period between 2004/05 and 2010/1 1 8 and teacher data collected between 
2008/09 and 2010/11 were analyzed to examine stability in school scores. Only three years 
of teacher data were used because the staff survey was modified substantially after the 
2007/08 academic year. More details on the sample used for these analyses are provided in 
box 2 in the main report. 


Are the school climate measures related to student achievement and discipline infrac- 
tions ? Research question 4 focuses on predictive validity of school-level climate measures 
from the Cal-SCHLS surveys. Correlations were estimated between school-level school 
climate measures and school-level measures of academic achievement and suspensions (see 
box 2 in the main report). These analyses were based on student and staff survey data 
collected in 2008/09 and 2010/11. 
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Sensitivity analyses. The study team conducted two sets of sensitivity analyses to deter- 
mine whether low response rates bias the results. First, school climate scores, 9 student 
demographic characteristics, staff characteristics, and other school characteristics were 
compared between two groups of schools: those with low response rates (lower than 
60 percent on the student survey and lower than 50 percent on the staff survey) and those 
with high response rates (rates above those levels). The response rate was calculated by 
dividing the number of surveys returned by grade 7 enrollment (student survey) or number 
of teachers in the school (staff survey). Second, differences by response rates were examined 
for the measurement model parameters (research question 1), respondent- and school-level 
reliabilities (research question 2), stability of school-level climate scores (research question 
3), and relationships of school climate measures to student achievement and discipline 
infractions (research question 4). The sensitivity analysis results are shown in appendix E. 
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Appendix D. Detailed results 


A detailed description of the results for research questions 1 and 3 is presented below. 

What school climate domains in middle school do the survey items measure?? 

Student survey exploratory factor analysis. Exploratory factor analysis (EFA) models were 
estimated to determine the number of factors underlying the data and the measurement 
structure of the latent factors. Using a randomly selected sample of approximately half the 
schools from the full sample, models were estimated for 10'factor solutions with between 
1 and 10 factors. All models with two or more factors met the root mean square error of 
approximation (RMSEA) criterion for good fit (.06 or lower; table Dl). All models with 
three or more factors met the standardized root mean square residual (SRMR) criterion for 
good fit (.08 or lower). All models with five or more factors met the comparative fit index 
(CFI) and Tucker-Lewis index (TLI) criterion for goodness of fit (.95 or higher). Thus, the 
fiveTactor model was retained as the preferred model and used as the starting point for 
comparative factor analysis (CFA) models. 

Box Dl defines the terms used in describing the results. 

The item factor loadings from the fiveTactor model suggested the following constructs, 
with items grouped based on the factor onto which they loaded most strongly: safety and 
connectedness (items 11-15 and item 119), caring relationships with teachers (items 16-21), 
meaningful participation (items 22-24), substance use at school and delinquency (items 
55-75, 102, and 107-112), and bullying and discrimination (items 100 and 101, 103-106, 
113-118, and 120). Item 22 (“At school, I do interesting activities”) was estimated to load 


Table Dl. Exploratory factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 3 

Tucker-Lewis 

index 0 

1 

.061 

.155 

.416 

.384 

2 

.037 

.101 

.796 

.773 

3 

.027 

.050 

.894 

.876 

4 

.021 

.043 

.941 

.927 

5 d 

.017 

.034 

.963 

.951 

6 

.013 

.027 

.979 

.971 

7 

.011 

.019 

.987 

.980 

8 

.010 

.017 

.990 

.985 

9 

.008 

.013 

.994 

.990 

10 

.007 

.011 

.996 

.992 


Note: The analytic sample consists of 106,928 students in 464 middle schools that administered the survey 
in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Box Dl. Definitions of terms used in describing the results 


Factor. The underlying, unobserved variable explained by variation in responses on observed 
survey items. 

Construct. General, subjective label applied to a factor to describe what it is. 

Summary measure. Respondent-level, average of responses on the items that comprise a 
construct. 

Summary score. School average of respondent-level summary measures. 


almost equally onto safety and connectedness, caring relationships with adults, and mean- 
ingful participation but was included with meaningful participation for conceptual clarity. 
The estimated item factor loadings for the five-factor EFA model are shown in appendix F. 

Student survey comparative factor analysis. The five-factor EFA solution was used as a 
starting point for a series of nested CFA models using data from a sample that included 
only the schools that were not selected for the EFA analytic sample. Five CFA models were 
tested to determine which factor specification had the best fit with the data (table D2). 

Model 1 specified the five-factor measurement structure suggested by the preferred EFA 
model, resulting in good fit based on RMSEA. Model 2 separated the items from the 
substance use at school and delinquency factors into two factors — one that included the 
substance use item and one for the delinquency items, as suggested by the factor loadings 
in the EFA for seven factors. This specification improved model fit based on all indices. 
Model 3 separated the safety and connectedness factor into two factors, guided by the 
factor loadings in the eight-factor EFA, resulting in improved model fit. Model 4 split the 
bullying items and discrimination items off from their original single factor, guided by 
factor loadings in the nine-factor EFA, further improving model fit. Model 5 moved item 
118 (“During the past 12 months, how often were you harassed or bullied for any other 
reason?”) from the bullying factor to the discrimination factor, as suggested by the 10-factor 
EFA results, weakening the model fit. Fit indices showed model 4 to be the preferred model, 
with eight constructs: safety, connectedness, caring relationships with adults, meaningful 
participation, substance use at school, delinquency, bullying, and discrimination. 

Correlations were estimated between these eight latent factors to determine whether they 
were sufficiently distinct from one another to justify including them as different factors. 
The results revealed high correlations for safety and connectedness (r = .81), bullying and 
discrimination (r = .82), and substance use and delinquency (r = .74). Correlations for safety 
and connectedness and for bullying and discrimination were high enough that a Fornell 
and Larcker (1981) test did not support discriminant validity. The two factors in each pair 
were combined, which resulted in a six-factor solution identical to that specified in model 
2. Thus, model 2 was the selected model and included the following constructs, which can 
be understood as the domains of school climate (shown in figure 1 in the main report): 
safety and connectedness, caring relationships with adults, meaningful participation, sub- 
stance use at school, bullying and discrimination, and delinquency. These six constructs 
serve as the basis for the reliability and validity analyses addressed in subsequent research 
questions. The factor loadings for each survey item on its respective construct are shown in 
table D3. Factor correlations are shown in table D4- 
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Table D2. Comparative factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey 


Model 

Factors 

Root mean 
square error of 
approximation 3 

Weighted root 
mean square 
residual b 

Comparative 
fit index' 

Tucker-Lewis 

index' 

1 

Five-factors, consistent 
with benchmark exploratory 
factor analysis 

.028 

10.048 

.906 

.899 

2 

Six-factors, separates 
substance use/delinquency d 

.025 

8.706 

.926 

.920 

3 

Seven-factors, separates 
connectedness/safety 

.023 

8.002 

.936 

.931 

4 

Eight-factors, separates 
bullying/discrimination e 

.022 

7.469 

.941 

.936 

5 

Eight-factors, qll8 loads on 
discrimination factor 

.022 

7.524 

.940 

.935 


Note: The analytic sample consists of 107,260 students in 485 middle schools that administered the survey 
in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Selected model based on Fornell and Larcker’s (1981) discriminant validity test. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


Table D3. Factor loadings for the selected confirmatory factor analysis model 
(model 2) for the student survey of the California School Climate, Health, and 
Learning Survey 


Item 

number 

Description 

Estimated loadings 

Standardized 

loadings 

Safety and connectedness 

11 

1 feel close to people at this school 

1.00 

0.52 

12 

1 am happy to be at this school 

1.45 

0.76 

13 

1 feel like 1 am a part of this school 

1.47 

0.77 

14 

Teachers at school treat students fairly 

1.32 

0.69 

15 

1 feel safe in my school 

1.49 

0.78 

119 

Flow safe do you feel at school 

1.31 

0.68 

Caring relationships with adults 

16 

Adult who really cares about me 

1.00 

0.74 

17 

... who tells me when 1 do a good job 

1.08 

0.80 

18 

... who notices when 1 am not there 

0.93 

0.69 

19 

... who always wants me to do my best 

1.10 

0.82 

20 

... who listens to me when 1 have something to say 

1.11 

0.82 

21 

... who believes that 1 will be a success 

1.12 

0.83 

Meaningful participation 

22 

1 do interesting activities 

1.00 

0.73 

23 

1 help decide things like activities or rules 

1.00 

0.73 

24 

1 do things that make a difference 

1.05 

0.76 

(continued) 
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Item 

number 

Description 

Estimated loadings 

Standardized 

loadings 

Substance use at school 

55 

Lifetime, been drunk or high at school 

1.00 

0.92 

72 

Past 30 days, smoked cigarettes at school 

1.03 

0.95 

73 

... drank alcohol at school 

0.95 

0.87 

74 

... smoked marijuana at school 

1.04 

0.96 

75 

... used other illegal drug at school 

0.99 

0.91 

Bullying and discrimination 

100 

Past 12 months, been pushed or shoved 

1.00 

0.68 

101 

... been afraid of being beaten up 

0.91 

0.62 

103 

... had rumors spread about you 

1.04 

0.71 

104 

. . . had sexual jokes made to you 

1.05 

0.72 

105 

... been made fun of because of looks 

1.02 

0.69 

106 

... had your property stolen or damaged 

0.99 

0.68 

113 

... been bullied because of race/ethnicity 

1.01 

0.69 

114 

... been bullied because of religion 

0.96 

0.66 

115 

... been bullied because of gender 

1.06 

0.72 

116 

... been bullied because you are LGBT 

1.07 

0.73 

117 

... been bullied because of disability 

1.05 

0.72 

118 

... been bullied for any other reason 

1.05 

0.72 

120 

... been cyber-bullied 

0.91 

0.62 

Delinquency 

102 

Past 12 months, been in a physical fight 

1.00 

0.73 

107 

... been offered or sold an illegal drug 

1.10 

0.80 

108 

... damaged school property on purpose 

1.09 

0.79 

109 

... carried a gun 

1.18 

0.86 

110 

... carried any other weapon 

1.17 

0.85 

111 

... been threatened or injured by a weapon 

1.15 

0.84 

112 

... seen someone carrying a weapon 

1.04 

0.75 


Note: The analytic sample consists of 107,260 students in 485 middle schools that administered the survey 
in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


Table D4. Correlations of the school climate measures for the student survey of the 

California School Climate, Health, and Learning Survey 




Safety and 
connectedness 

Caring 
relationships 
with adults 

Meaningful 

participation 

Substance 
use at school 

Bullying and 
discrimination 

Caring relationships with adults 

.60 





Meaningful participation 

.54 

.61 




Substance use at school 

-.35 

-.28 

-.16 



Bullying and discrimination 

-.37 

-.16 

-.10 

.38 


Delinquency 

-.41 

-.28 

-.17 

.74 

.65 


Note: The analytic sample consists of 107,260 students in 485 middle schools who provided responses on 
the 2008/09 or 2009/10 survey. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Staff survey exploratory factor analysis. As with the student survey items, EFA models 
were estimated for the staff survey data to determine the number of underlying factors and 
the measurement structure of the latent factors. Fit indices were estimated for 10 models 
with 1-10 factors using a randomly selected sample of approximately half the schools from 
the full sample. Models with 5, 9, and 10 factors could not converge (table D5). Of the 
estimated models, the four-factor model and the models with six or more factors met the 
RMSEA criterion of good fit; only the eight-factor model met the CFI criterion for good fit. 

The eight'factor model was the preferred model that met most rule-of-thumb criteria 
for good fit. However, an investigation of item factor loadings for the eight-factor model 
revealed several items that cross-loaded onto multiple factors. Thus, a simpler model that 
included six factors was retained as the starting point for the CFA models. 

The item factor loadings from the six-factor model suggested the following constructs, 
with items grouped based on the factor onto which they loaded most strongly: support 
and safety (items 6-25 and 29-32); school discipline practices (items 26-28); caring 
relationships among staff and between staff and students (items 33-42); professional 
development needs around academic, socioemotional, and cultural issues (items 43-52); 
student health and engagement (items 53-56); and student delinquency (items 57-69). 
The estimated item factor loadings for the selected six-factor EFA model are shown in 
appendix G. 

Staff survey comparative factor analysis. Using the six-factor EFA solution as a starting 
point, 16 CFA models were estimated to determine fit indices (table D6). Model 1 speci- 
fied the six-factor measurement structure suggested by the preferred EFA, resulting in good 


Table D5. Exploratory factor analysis goodness-of-fit statistics for the staff survey 
of the California School Climate, Health, and Learning Survey 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 3 

Tucker-Lewis 

index 3 

1 

.096 

.152 

.624 

.611 

2 

.082 

.123 

.733 

.715 

3 

.073 

.087 

.797 

.776 

4 

.060 

.053 

.867 

.848 

5 


No convergence 


6 d 

.048 

.038 

.919 

.901 

7 

.044 

.033 

.937 

.920 

8 e 

.039 

.028 

.950 

.935 

9 


No convergence 


10 


No convergence 



Note: The analytic sample consists of 7,079 teachers in 393 middle schools that administered the survey in 
2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Selected model. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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fit based on RMSEA. Model 2 split items gauging appreciation for diversity, which cross- 
loaded on a separate factor in the eight-factor EFA, away from the support and safety con- 
struct, resulting in improved model fit. Model 3 moved items 29 (“This school is safe place 
for students”) and 30 (“This school is safe place for staff’) from the support and safety 
construct to the discipline practices construct, as they cross-loaded on the two constructs, 
resulting in better fit. Models 4, 5, and 6 experimented with different means of separating 
the new support construct, but none improved model fit. 

In models 7-13, items from the support factor were iteratively parsed into separate con- 
structs representing meaningful participation, provision of resources for staff, student 
supports, and staff supports. Items from the discipline factor were separated to create a 
new safety factor, and items from the relationships factor were separated into caring staff- 
student relationships and staff-peer relationships. Professional development items were 
split into three factors, each representing a different area of competence. The addition of 
each new factor resulted in improved model fit. Model 13 was the preferred model based 
on fit indices, with 14 constructs: student supports, staff supports, provision of resources, 
student participation, safety, discipline practices, appreciation for diversity, staff-student 
relationships, staff-peer relationships, academic professional development needs, socio- 
emotional professional development needs, cultural professional development needs, 
student health and engagement connectedness, and student delinquency. 

Correlations were estimated between these 14 latent factors to determine whether they 
were sufficiently distinct from one another. The factor correlations indicated high correla- 
tions between the following pairs of factors: 

• Student supports and student participation (r = .88). 

• Student supports and safety (r = .85). 

• Appreciation for diversity and student participation (r = .87). 

• Cultural professional development needs and socioemotional professional develop- 
ment needs (r = .88). 

The strength of these correlations did not support discriminant validity based on a Fornell 
and Larcker (1981) test. The model was thus reduced by combining items from the aca- 
demic supports, meaningful student participation, and safety factors and the cultural and 
socioemotional professional development needs factors. A new CFA model with 11 factors 
(model 14 in table D6) was then estimated, which resulted in factor correlations between 
the new student supports factor and the appreciation for diversity (r = .84) and discipline 
factors (r = .80) and between the two professional development needs factors (r = .85). 
Thus, these factors were combined, resulting in a new eight-factor CFA. 

In the eight-factor CFA (model 15), the correlation between the new student supports 
factor and the staff supports factor (r = .81) did not support discriminant validity; thus, 
these were combined into a seven-factor model (model 16), the model that was ultimate- 
ly selected. The seven constructs identified in the selected model (shown in figure 2 in 
the main report) were support and safety, caring staff-student relationships, staff-peer 
relationships, professional development needs, student health and engagement, student 
delinquency, and resource provision. These seven constructs serve as the basis for the 
reliability and validity analyses addressed in subsequent research questions. The factor 
loadings for each survey item are shown in table D7. The factor correlations are shown in 
table D8. 
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Table D6. Comparative factor analysis goodness-of-fit statistics for the staff survey 
of the California School Climate, Health, and Learning Survey 

Model 

Factors 

Root mean 
square error of 
approximation 11 

Weighted root 
mean square 
residual b 

Comparative 
fit index c 

Tucker-Lewis 

index 0 

1 

6 factors, consistent with baseline 
exploratory factor analysis 

.041 

4.503 

.913 

.910 

2 

7 factors, separates supports/diversity 

.039 

4.218 

.922 

.919 

3 

7 factors, q29 q30 with discipline 

.038 

4.085 

.927 

.924 

4 

8 factors, separates student supports/ 
staff supports 

.041 

4.430 

.913 

.908 

5 

8 factors, separates supports/staff 
relationships 

.039 

4.205 

.921 

.918 

6 

9 factors, separates student/staff 
supports/staff resources 

.038 

4.064 

.925 

.921 

7 

8 factors, separates supports/ 
participation 

.037 

3.962 

.930 

.927 

8 

9 factors, separates safety/discipline 

.036 

3.888 

.932 

.929 

9 

10 factors, separates support/staff 
resources 

.035 

3.728 

.936 

.933 

10 

11 factors, separates student 
support/staff support 

.034 

3.540 

.941 

.938 

11 

12 factors, separates student 
relations/staff relations 

.031 

3.197 

.951 

.948 

12 

13 factors, separates relationships/ 
high expectations 


Could not be estimated 


13 

14 factors, separates academic/ 
cultural/social professional 
development 11 

.030 

2.887 

.956 

.953 

14 

11 factors, combined student support/ 
safety/participation and cultural/ 
social professional development 

.033 

3.426 

.943 

.940 

15 

8 factors, combined student support/ 
diversity/discipline and academic/ 
sociocultural professional development 

.039 

4.261 

.920 

.917 

16 

7 factors, combined student support/ 
staff collegiality 6 

.041 

4.520 

.912 

.908 


Note: The analytic sample consists of 7,358 teachers in 371 middle schools that administered the survey in 
2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Preferred model. 

e. Selected model based on Fornell and Larcker's (1981) discriminant validity test. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table D7. Factor loadings for the selected confirmatory factor analysis model 
(model 16) for the staff survey of the California School Climate, Health, and 
Learning Survey 


Item 

number Description 


Support and safety 

6 This school is a supportive place for students to learn 1.00 0.87 


7 

... sets high standards for academic performance 

0.94 

0.82 

8 

... promotes academic success for all students 

1.02 

0.88 

9 

... emphasizes helping students academically 

0.96 

0.84 

10 

... provides adequate support services for students 

0.78 

0.68 

11 

... emphasizes teaching lessons in relevant ways 

0.89 

0.78 

12 

... is a supportive and inviting place for staff to work 

1.02 

0.89 

13 

... promotes trust and collegiality among staff 

1.00 

0.87 

16 

... provides opportunities for student decisionmaking 

0.74 

0.64 

17 

... gives students class discussion opportunities 

0.90 

0.78 

18 

... gives students extracurricular activity opportunities 

0.83 

0.72 

19 

... gives students opportunities to make a difference 

0.82 

0.71 

20 

... encourages students to enroll in rigorous courses 

0.84 

0.73 

21 

... emphasizes culturally relevant instruction 

0.79 

0.69 

22 

... has staff examine their cultural biases through P.D. 

0.73 

0.64 

23 

... prioritizes closing the racial achievement gap 

0.77 

0.67 

24 

... fosters appreciation for student diversity 

0.98 

0.86 

25 

... emphasizes showing respect for student cultures 

0.96 

0.84 

26 

... clearly communicates rules to students 

0.94 

0.82 

27 

... handles discipline problems fairly 

1.01 

0.88 

28 

... effectively handles discipline and behavior problems 

1.01 

0.88 

29 

... is a safe place for students 

0.99 

0.87 

30 

... is a safe place for staff 

0.98 

0.85 

31 

... is welcoming to and facilitates parent involvement 

0.93 

0.81 

32 

... has clean and well-maintained facilities 

0.77 

0.67 

Caring staff-student relationships 

33 

Adults in this school really care about every student 

1.00 

0.88 

34 

... acknowledge and pay attention to students 

1.05 

0.93 

35 

. . . want every student to do their best 

1.03 

0.91 

36 

... listen to what students have to say 

1.04 

0.92 

37 

... believe that every student can be a success 

1.01 

0.90 

38 

... treat all students fairly 

1.06 

0.94 

39 

... treat every student with respect 

1.07 

0.94 

Staff- 

-peer relationships 



40 

... have close professional relationships with each other 

1.00 

0.87 

41 

... support and treat each other with respect 

1.06 

0.92 

42 

... feel a responsibility to improve this school 

1.03 

0.90 

Professional development needs 

43 

1 need training in meeting academic standards 

1.00 

0.84 

44 

... evidence-based methods of instruction 

0.96 

0.81 

45 

... positive behavioral supports/classroom management 

0.97 

0.81 

46 

... working with diverse racial and cultural groups 

1.04 

0.87 


Estimated Standardized 

loadings loadings 


(continued) 


D-8 







Table D7. Factor loadings for the selected confirmatory factor analysis model 
(model 16) for the staff survey of the California School Climate, Health, and 
Learning Survey (continued) 


Item 

number 

Description 

Estimated 

loadings 

Standardized 

loadings 

47 

... culturally relevant pedagogy for students 

1.05 

0.88 

48 

... serving English language learners 

0.97 

0.82 

49 

... closing the achievement gap 

0.96 

0.81 

50 

... serving special education students 

0.93 

0.78 

51 

... meeting the social and emotional needs of students 

0.97 

0.82 

52 

... creating a positive school environment 

1.05 

0.88 

Student health and engagement 

53 

Students at this school are healthy and physically fit 

1.00 

0.74 

54 

... arrive at school alert and rested 

1.13 

0.84 

55 

... are motivated to learn 

1.22 

0.91 

56 

... are well-behaved 

1.18 

0.87 

Student delinquency 

57 

Student alcohol and drug use a problem at school 

1.00 

0.76 

58 

Student tobacco use... 

0.92 

0.70 

59 

Harassment and bullying among students... 

1.03 

0.78 

60 

Physical fighting between students... 

1.09 

0.83 

61 

Disruptive student behavior... 

1.11 

0.85 

62 

Racial conflict among students... 

0.94 

0.72 

63 

Student depression or mental health... 

0.79 

0.60 

64 

Lack of respect for staff by students... 

1.10 

0.84 

65 

Cutting classes or being truant. . . 

1.02 

0.78 

66 

Gang-related activity... 

0.99 

0.76 

67 

Weapons possession... 

1.02 

0.78 

68 

Vandalism... 

1.00 

0.76 

69 

Theft... 

0.98 

0.75 

Resource provision 

14 

. . . provides resources for staff to do their jobs well 

1.00 

0.89 

15 

... provides resources to work with special education (IEP) students 

0.99 

0.88 


Note: The analytic sample consists of 7,358 teachers in 371 middle schools that administered the survey in 
2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 


Table D8. Correlations of staff survey measures on the California School Climate, Health, and Learning 
Survey 

Measure 

Support 
and safety 

Caring 

staff-student 

relationships 

Staff-peer 

relationships 

Professional 

development 

needs 

Student 
health and 
engagement 

Student 

delinquency 

Caring staff-student relationships 

.62 






Staff-peer relationships 

.66 

.83 





Professional development needs 

-.16 

-.16 

-.12 




Student health and engagement 

.51 

.44 

.41 

-.13 



Student delinquency 

-.50 

-.39 

-.36 

.20 

-.63 


Resource provision 

.75 

.44 

.52 

-.14 

.39 

.38 

Note: The analytic sample consists of 7,358 teachers in 371 middle schools that administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table D9. Stability of student survey measures on the California School Climate, Health, and Learning 
Survey for one-year intervals, by survey year (correlations) 


Measure 

2004/05 to 
2005/06 

2005/06 to 
2006/07 

2006/07 to 
2007/08 

2007/08 to 
2008/09 

2008/09 to 
2009/10 

2009/10 to 
2010/11 

One-year 

average 

Safety and connectedness 

.80 

.65 

.77 

.83 

.82 

.91 

.80 

Caring relationships with adults 

.26 

.31 

.67 

.68 

.61 

.75 

.52 

Opportunities for participation 

.37 

.07 

.73 

.67 

.65 

.77 

.54 

Substance use at school 

.51 

.20 

.10 

.38 

.29 

.58 

.36 

Bullying and discrimination 

.48 

.50 

.52 

.44 

.40 

.62 

.49 

Delinquency 

.69 

.41 

.57 

.56 

.60 

.75 

.61 

Number of one-year survey intervals 

64 

34 

40 

38 

38 

41 

na 


na is not applicable. 

Note: The analytic sample consists of 129 schools that administered the survey in consecutive years, for a total of 255 intervals. 
Source: 2004/05 to 2010/11 Cal-SCHLS student survey. 


How stable are the school-level climate measures? 

To examine stability of schooUevel climate summary scores over time, schooUevel aggre- 
gate scores were correlated at one- and two-year intervals of survey administration. As 
described in appendix C, correlations were estimated for all one- and two-year intervals of 
survey administration. Results are presented for the student survey and the teacher survey. 

Student survey. The average correlation of school-level summary scores over one-year 
intervals ranged from .36 (substance use) to .80 (safety and connectedness; table D9). 
These averages were weighted by the number of schools that administered the student 
survey at each one-year interval (see appendix C). The one-year stability of the scores gen- 
erally improved over time. Between the 2009/10 and 2010/11 administrations of the survey, 
the stability correlations for all scores were at least .58 (for substance use) and above .70 for 
four of the constructs. 

At two-year intervals the average stability of all student survey scores was diminished 
(table DIO). For example, whereas safety and connectedness was highly correlated from 
year to year (r = .80), the average correlation is only .68 at two-year intervals. The decrease 
in stability is largest for safety and connectedness. Safety and connectedness was the most 
stable of the school-level scores over time, followed by delinquency, meaningful participa- 
tion, caring relationships with adults, bullying and discrimination, and substance use at 
school. 

Staff survey. For the staff survey school-level summary scores, the weighted average cor- 
relation over one-year intervals ranged from -.07 to .78, with professional development 
needs the outlier (-.07); all other scores had averages of .56 or higher (table Dll). That 
the average correlation for professional development needs was negative suggests that it 
is not a reliable measure of an enduring school characteristic. The two-year correlations, 
between the 2008/09 and 2010/11 administration of the teacher survey, were between .38 
and .66. Student health and engagement and student delinquency had one-year average 
correlations and two-year correlations above .60. 
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Table DIO. Stability of student survey measures on the California School Climate, Health, and Learning 
Survey for two-year intervals, by survey year (correlations) 


Measure 

2004/05 to 
2006/07 

2005/06 to 
2007/08 

2006/07 to 
2008/09 

2007/08 to 
2009/10 

2008/09 to 
2010/11 

Two-year 

average 

Safety and connectedness 

.65 

.69 

.60 

.73 

.62 

.68 

Caring relationships with adults 

.35 

.48 

.44 

.46 

.44 

.45 

Meaningful participation 

.33 

.52 

.45 

.59 

.57 

.51 

Substance use at school 

.19 

.35 

.32 

.39 

.38 

.34 

Bullying and discrimination 

.48 

.43 

.34 

.38 

.44 

.41 

Delinquency 

.49 

.58 

.51 

.61 

.51 

.56 

Number of two-year survey intervals 

269 

585 

259 

637 

191 

— 


Note: The analytic sample consists of 959 schools that administered the survey in two-year intervals, for a total of 1,941 two-year 
intervals. 

Source: 2004/05 to 2010/11 Cal-SCHLS student survey. 


Table Dll. Stability of staff survey measures on the California School Climate, 
Health, and Learning Survey, by survey intervals (correlations) 


Measure 

2008/09 

One-year 
interval 
2009/10 to 
2010/11 

One-year 

average 

Two-year 
interval 
2008/09 to 
2010/11 

Support and safety 

.71 

.53 

.62 

.42 

Caring staff-student relationships 

.59 

.80 

.69 

.45 

Staff-peer relationships 

.73 

.83 

.78 

.38 

Professional development needs 

.25 

-.43 

-.07 

.43 

Student health and engagement 

.73 

.71 

.72 

.66 

Student delinquency 

.54 

.86 

.69 

.64 

Resource provision 

.56 

.57 

.56 

.42 

Number of survey intervals 

22 

20 

42 

119 


Note: The analytic sample consists of 34 schools that administered the survey in consecutive years and 119 
schools that administered the survey at two-year intervals. 

Source: 2008/09 to 2010/11 Cal-SCHLS staff survey. 
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Appendix E. Sensitivity analyses 


Student and staff survey response rates vary substantially across schools. Because the risk 
of biased results rises as the response rate drops, analyses were conducted to determine 
whether schools with low student or staff response rates differed from other schools in 
climate, staff, student demographics, or other characteristics. The team also conducted 
sensitivity analyses to determine whether low response rates biased the results. 

Differences in schools by survey response rate 

Student and staff survey response rates — the proportion of eligible individuals who com- 
pleted the surveys — varied substantially across schools in the analytic samples. The student 
response rate for each school was calculated by dividing the number of student surveys 
returned by grade 7 enrollment. For staff the response rate was calculated by dividing the 
number of staff surveys completed by the number of eligible respondents, as reported by 
each school prior to survey administration. All school staff — teachers, administrators, and 
other certificated and classified staff — are asked to complete the CahSCHLS staff survey, 
but only teacher survey responses were examined in this report. While the student survey 
was administered only in grade 7, the staff survey included all middle school teachers, so 
the response rate is based on all eligible staff at each middle school. 

For 2008/09-2009/10 the median survey response rate for the student survey was 
74.6 percent. One-quarter of schools had a student response rate below 60 percent, 
and 10 percent had a rate below 35 percent. The estimated response rates for the staff 
survey were substantially lower. The median response rate for middle school teachers in 
2008/09-2009/10 was 48 percent, with 25 percent of schools having a response rate below 
28 percent. 

School climate summary scores, student demographic characteristics, staff characteris- 
tics, and other school characteristics were compared between schools with low response 
rates (lower than 60 percent on the student survey and lower than 50 percent on the staff 
survey) and those with high response rates. 

Schools with high response rates on the student survey had significantly higher average 
scores on the substance use survey construct than did schools with low response rates 
(table El). There were no significant differences between school groups on other survey 
constructs. Also, schools with high response rates had significantly lower proportions of 
Black and low-income students and suspension rates and significantly higher academic 
performance than did schools with low response rates. 

Schools with high response rates on the teacher survey had higher scores on all but two 
of the teacher survey constructs (table E2). The schools with high response rates exhibited 
higher scores on support and safety, caring staff-student relationships, staff-peer relation- 
ships, student health and engagement, and resource provision. Schools with high response 
rates also had higher percentages of White students, higher academic performance, and 
lower percentages of Hispanic students and low-income students. 

These differences by response-rate group indicate that analyzing data only from schools 
with high response rates on the student and staff surveys could reduce the generalizabil- 
ity of the results to other schools in the state, since schools with high response rates are 
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Table El. Comparison of student survey measures on the California School Climate, 
Health, and Learning Survey, by high and low response rates 


School characteristic 

All sample schools 

Standard 
Mean deviation 

Schools with 
high student 
response rate 
(at least 
60 percent) 

Schools with 
low student 
response rate 
(less than 
60 percent) 

Difference 

(standard 

deviation) 

School climate scores 

Safety and connectedness 

300 

50 

300 

299 

+0.03 

Caring relationships with adults 

300 

50 

299 

302 

-0.05 

Meaningful participation 

300 

50 

300 

299 

+0.03 

Substance use at school 

300 

50 

302 

294 

+0.16** 

Bullying and discrimination 

300 

50 

299 

304 

-0.11 

Delinquency 

300 

50 

300 

300 

0.00 

Demographics, percentage of students who are 

Asian 

9 

14 

10 

8 

+0.11 

Black 

6 

8 

6 

9 

-0.41** 

Hispanic 

47 

28 

47 

49 

-0.07 

White 

30 

25 

31 

28 

+0.10 

Low-income 

53 

28 

51 

58 

-0.26** 

Other school characteristics 

Academic performance index 

777 

83 

783 

770 

+0.26** 

English language arts test score 

358 

26 

360 

353 

+0.27** 

average (grade 7) 






English language arts test score 

358 

26 

360 

352 

+0.29** 

average (grade 8) 






Math test score average 

354 

29 

355 

350 

+0.18** 

Suspensions per 100 students 

25 

24 

23 

30 

-0.27** 


♦♦Difference is significant at the p < .05 level based on a two-sample t-test with unequal variances. 

Note: The analytic sample consists of 949 middle schools (720 high-response-rate schools and 229 low- 
response-rate schools) that administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


different from schools with low response rates. Generalizability to all schools in the state is 
best ensured by including all schools in the analytic sample. However, the extent to which 
schools with low response rates exhibit greater within-school differences between respon- 
dents and nonrespondents may reduce generalizability. The comparisons do not provide 
information on the differences between respondents and nonrespondents in schools. 

Replication in schools with low and high survey response rates 

W hat school climate domains in middle school do the survey items measure? All analy- 
ses of the student survey data related to research question 1 were replicated for two sub- 
samples: students in schools with student survey response rates of 60 percent or greater and 
students in schools with rates below 60 percent. The model fit estimates for these subgroup 
analyses are shown. The same decisionmaking criteria were used as for the full sample 
analysis. There were no substantive differences in conclusions to research question 1 in 
either of the two subsamples. The same measurement structure was identified for students 
in schools with high and low survey response rates (tables E3-E6). 
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Table E2. Comparison of staff survey measures on the California School Climate, Health, and Learning 
Survey, by high and low response rates 



All sample schools 

Schools with 

Schools with 


School characteristic 

Mean 

Standard 

deviation 

high teacher 
response 
rate (at least 
50 percent) 

low teacher 
response rate 
(less than 
50 percent) 

Difference 

(standard 

deviation) 

School climate scores 

Support and safety 

300 

50 

308 

295 

+0.30** 

Caring staff-student relationships 

300 

50 

306 

295 

+0.24** 

Staff-peer relationships 

300 

50 

308 

294 

+0.30** 

Professional development needs 

300 

50 

296 

301 

-0.13 

Student health and engagement 

300 

50 

304 

297 

+0.16** 

Student delinquency 

300 

50 

296 

304 

-0.16** 

Staff resources 

300 

50 

309 

294 

+0.33** 

Demographics, percentage of students who are 

Asian 

9 

14 

9 

10 

-0.10 

Black 

6 

7 

5 

6 

-0.07 

Hispanic 

47 

28 

44 

49 

-0.15** 

White 

32 

24 

35 

29 

+0.26** 

Low-income 

50 

28 

48 

52 

-0.14** 

Other school characteristics 

Academic performance index 

787 

82 

794 

780 

+0.17** 

English language arts test score average (grade 7) 

361 

26 

362 

359 

+0.12 

English language arts test score average (grade 8) 

361 

27 

363 

358 

+0.12 

Math test score average 

356 

29 

358 

354 

+0.13 

Suspensions per 100 students 

23 

21 

22 

24 

-0.13 


“Difference is significant at the p < .05 level based on a two-sample t-test with unequal variances. 

Note: The analytic sample consists of 762 middle schools (362 high-response-rate schools and 400 low-response-rate schools) that 
administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table E3. Exploratory factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey, schools with 
high response rates 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

.067 

.155 

.337 

.302 

2 

.041 

.101 

.761 

.735 

3 

.031 

.050 

.871 

.848 

4 

.024 

.042 

.930 

.912 

5 d 

.019 

.034 

.956 

.942 

6 

.015 

.027 

.974 

.964 

7 

.012 

.019 

.984 

.976 

8 

.011 

.017 

.988 

.981 

9 

.009 

.013 

.993 

.988 

10 

.008 

.011 

.995 

.990 


Note: The analytic sample consists of 184,509 students in 720 middle schools with response rates of 60 per- 
cent or higher that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


Table E4. Exploratory factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey, schools with 
low response rates 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

.076 

.165 

.556 

.532 

2 

.050 

.108 

.817 

.796 

3 

.041 

.054 

.885 

.864 

4 

.033 

.045 

.930 

.913 

5 a 

.026 

.037 

.957 

.944 

6 

.022 

.028 

.972 

.961 

7 

.019 

.021 

.981 

.972 

8 

.015 

.018 

.989 

.982 

9 

.012 

.013 

.993 

.988 

10 

.011 

.012 

.994 

.990 


Note: The analytic sample consists of 29,679 students in 229 middle schools with response rates below 
60 percent that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Table E5. Comparative factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey, schools with 
high response rates 


Model 

Factors 

Root mean 
square error of 
approximation 3 

Weighted root 
mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

Five-factors, consistent 
with benchmark exploratory 
factor analysis 

12.495 

.025 

.910 

.904 

2 

Six-factors, separates 
substance use/delinquency d 

10.727 

.022 

.931 

.926 

3 

Seven-factors, separates 
connectedness/safety 

9.954 

.020 

.940 

.935 

4 

Eight-factors, separates 
bullying/discrimination e 

9.192 

.019 

.948 

.943 

5 

Eight-factors, qll8 loads 
on discrimination factor 

9.293 

.019 

.946 

.941 


Note: The analytic sample consists of 184,509 students in 720 middle schools with response rates of 60 per- 
cent or higher that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Selected model based on Fornell and Larcker’s (1981) discriminant validity test. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 


Table E6. Comparative factor analysis goodness-of-fit statistics for the student 
survey of the California School Climate, Health, and Learning Survey, schools with 
low response rates 


Model 

Factors 

Root mean 
square error of 
approximation 3 

Weighted root 
mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

Five-factors, consistent 
with benchmark exploratory 
factor analysis 

6.141 

.030 

.933 

.928 

2 

Six-factors, separates 
substance use/delinquency d 

5.271 

.027 

.947 

.943 

3 

Seven-factors, separates 
connectedness/safety 

4.775 

.024 

.956 

.952 

4 

Eight-factors, separates 
bullying/discrimination 6 

4.531 

.024 

.958 

.954 

5 

Eight-factors, qll8 loads 
on discrimination factor 

4.526 

.024 

.958 

.954 


Note: The analytic sample consists of 29,679 students in 229 middle schools with response rates below 
60 percent that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Selected model based on Fornell and Larcker's (1981) discriminant validity test. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCFILS student survey. 
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Similarly, the teacher models were reestimated for two subsamples: teachers in schools 
with staff survey response rates of 50 percent or greater and teachers in schools with rates 
below 50 percent. As with the sensitivity analyses for the student survey, there were no 
substantive differences in conclusions in either of the two subsamples. The same measure- 
ment structure was identified for teachers in schools with high and low survey response 
rates (tables E7-E10). 


Table E7. Exploratory factor analysis goodness-of-fit statistics for the staff survey 
of the California School Climate, Health, and Learning Survey, schools with high 
response rates 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 3 

Tucker-Lewis 

index 3 

1 

.096 

.152 

.499 

.482 

2 

.080 

.122 

.665 

.643 

3 

.071 

.085 

.741 

.714 

4 

.058 

.050 

.834 

.811 

5 

.053 

.043 

.869 

.845 

6 d 

.047 

.037 

.898 

.875 

7 

.043 

.033 

.920 

.898 

8 e 

.038 

.028 

.938 

.918 

9 


No convergence 


10 


No convergence 



Note: The analytic sample consists of 10,239 teachers in 362 middle schools with response rates of 50 per- 
cent or higher that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Selected model. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table E8. Exploratory factor analysis goodness-of-fit statistics for the staff survey 
of the California School Climate, Health, and Learning Survey, schools with low 
response rates 


Number of 
factors 

Root mean 
square error of 
approximation 3 

Standardized 
root mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

.101 

.152 

.637 

.625 

2 

.088 

.123 

.729 

.711 

3 

.075 

.985 

.811 

.791 

4 

.062 

.052 

.877 

.859 

5 


No convergence 


6 d 


No convergence 


7 

.044 

.032 

.943 

.928 

8 e 

.040 

.028 

.955 

.940 

9 

.036 

.025 

.964 

.951 

10 


No convergence 



Note: The analytic sample consists of 4,331 teachers in 400 middle schools with response rates below 
50 percent that administered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of .08 or lower. 

c. Recommended value of at least .95. 

d. Selected model. 

e. Preferred model. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table E9. Comparative factor analysis goodness-of-fit statistics for the staff survey of the California 
School Climate, Health, and Learning Survey Staff, schools with high response rates 


Model 

Factors 

Root mean 
square error of 
approximation 11 

Weighted root 
mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

6 factors, consistent with baseline 
exploratory factor analysis 

.040 

5.090 

.914 

.910 

2 

7 factors, separates supports/diversity 

.038 

4.771 

.922 

.919 

3 

7 factors, q29 q30 with discipline 

.037 

4.684 

.927 

.923 

7 

8 factors, separates supports/participation 

.036 

4.515 

.930 

.927 

11 

12 factors, separates student relations/ 
staff relations 

.030 

3.592 

.952 

.948 

13 

14 factors, separates academic/cultural/ 
social professional development 11 

.029 

3.260 

.956 

.953 

14 

11 factors, combined student support/ 
safety/participation and cultural/ 
social professional development 

.032 

3.835 

.944 

.941 

15 

8 factors, combined student support/ 
diversity/discipline and academic/ 
sociocultural professional development 

.037 

4.637 

.925 

.922 

16 

7 factors, combined student support/staff 
collegiality e 

.040 

5.112 

.913 

.909 


Note: The analytic sample consists of 10,239 teachers in 362 middle schools with response rates of 50 percent or higher that admin- 
istered the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Preferred model. 

e. Selected model based on Fornell and Larcker's (1981) discriminant validity test. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Table E10. Comparative factor analysis goodness-of-fit statistics for the staff survey of the California 
School Climate, Health, and Learning Survey, schools with low response rates 


Model 

Factors 

Root mean 
square error of 
approximation 3 

Weighted root 
mean square 
residual b 

Comparative 
fit index 0 

Tucker-Lewis 

index 0 

1 

6 factors, consistent with baseline 
exploratory factor analysis 

.046 

3.901 

.925 

.922 

3 

7 factors, q29 q30 with discipline 

.041 

3.463 

.939 

.937 

7 

8 factors, separates supports/participation 

.040 

3.362 

.942 

.939 

11 

12 factors, separates student relations/ 
staff relations 

.034 

2.708 

.960 

.957 

13 

14 factors, separates academic/cultural/ 
social professional development 

.032 

2.481 

.964 

.961 

14 

11 factors, combined student support/ 
safety/participation and cultural/ 
social professional development 

.037 

2.992 

.951 

.948 

16 

7 factors, combined student support/staff 
collegiality e 

.047 

3.964 

.923 

.919 


Note: The analytic sample consists of 4,331 teachers in 400 middle schools with response rates below 50 percent that administered 
the survey in 2008/09 or 2009/10. 

a. Recommended value of .06 or lower. 

b. Recommended value of 1.0 or lower or minimum value. 

c. Recommended value of at least .95. 

d. Preferred model. 

e. Selected model based on Fornell and Larcker's (1981) discriminant validity test. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 


How reliably do the surveys items measure school climate at the respondent and school 
levels? The student- and teacher-level internal consistency estimates of reliability were 
nearly identical between the subsamples with high and low response rates. There were no 
substantive differences between the two groups (tables Ell and El 2). 


Table Ell. Internal consistency reliability of the student survey measures on the 
California School Climate, Health, and Learning Survey Staff, by high and low 
response rates (Cronbach’s alpha) 


Measure 

High student response rate 
(at least 60 percent) 

Low student response rate 
(less than 60%) 

Safety and connectedness 

.80 

.79 

Caring relationships with adults 

.85 

.84 

Meaningful participation 

.68 

.68 

Substance use at school 

.88 

.87 

Bullying and discrimination 

.85 

.85 

Delinquency 

.83 

.82 


Note: Recommended value is at least .70. The analytic sample consists of 184,509 students in 720 schools 
with high response rates and 29,679 students in 229 schools with low response rates that administered the 
survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Table E12. Internal consistency reliability of staff survey measures on the California 
School Climate, Health, and Learning Survey Staff, by high and low response rates 
(Cronbach’s alpha) 


Measure 

High staff response rate 
(at least 50 percent) 

Low staff response rate 
(less than 50 percent) j 

Support and safety 

.96 

.96 

Caring staff-student relationships 

.95 

.94 

Staff-peer relationships 

.88 

.87 

Professional development needs 

.89 

.89 

Student health and engagement 

.84 

.83 

Student delinquency 

.91 

.91 

Resource provision 

.81 

.82 


Note: Recommended value is at least .70. The analytic sample consists of 10,239 teachers in 372 schools 
with high response rates and 4,331 teachers in 400 schools with low response rates that administered the 
survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 


The schooHevel reliabilities were mostly similar across the two samples as well, although 
the reliability coefficient for substance use on school property was noticeably lower for the 
subsample with low response rates (table E13). In the subsample with high response rates 
the k coefficient (Spearman-Brown reliability coefficient) for the substance use measure 
with a school sample size of 100 was .67, compared with .55 in the subsample with low 
response rates. This difference was due to the fact that the intraclass correlation for the 
substance use construct was smaller for the subsample with low response rates, imply- 
ing that there are fewer between-school differences in student substance use on school 


Table E13. School-level reliability of student survey measures on the California 
School Climate, Health, and Learning Survey, by high and low response rates 

Measure 

Intraclass 

correlation 


r given student sample size per school* 


226 

(mean) 

50 

100 

300 

400 

High response rate (at least 60 percent) 

Safety and connectedness 

.079 

.95 

.81 

.90 

.96 

.97 

Caring relationships with adults 

.027 

.86 

.58 

.74 

.89 

.92 

Meaningful participation 

.033 

.89 

.63 

.77 

.91 

.93 

Substance use at school 

.020 

.82 

.51 

.67 

.86 

.89 

Bullying and discrimination 

.020 

.82 

.51 

.67 

.86 

.89 

Delinquency 

.033 

.89 

.63 

.77 

.91 

.93 

Low response rate (less than 60 percent) 

Safety and connectedness 

.068 

.94 

.78 

.88 

.96 

.97 

Caring relationships with adults 

.027 

.86 

.58 

.74 

.89 

.92 

Meaningful participation 

.033 

.89 

.63 

.77 

.91 

.93 

Substance use at school 

.012 

.73 

.38 

.55 

.78 

.83 

Bullying and discrimination 

.016 

.79 

.45 

.62 

.83 

.87 

Delinquency 

.027 

.86 

.58 

.74 

.89 

.92 


Note: The analytic sample consists of 184,509 students in 720 schools with high response rates and 29,679 
students in 229 schools with low response rates that administered the survey in 2008/09 or 2009/10. 

a. Reliability predicted from Spearman-Brown formula; recommended value of at least .60. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Table E14. School-level reliability of staff survey measures on the California School 
Climate, Health, and Learning Survey, by high and low response rates 




r given teacher sample size per school 3 


Measure 

Intraclass 

correlation 

19 

(mean] 

5 

10 

30 

40 

High response rate (at least 50 percent) 

Support and safety 

.239 

.86 

.61 

.76 

.90 

.93 

Caring staff-student relationships 

.124 

.73 

.41 

.59 

.81 

.85 

Staff-peer relationships 

.149 

.77 

.47 

.64 

.84 

.88 

Professional development needs 

.045 

.47 

.19 

.32 

.59 

.65 

Student health and engagement 

.264 

.87 

.64 

.78 

.91 

.93 

Student delinquency 

.329 

.90 

.71 

.83 

.94 

.95 

Resource provision 

.118 

.72 

.40 

.57 

.80 

.84 

Low response rate (less than 50 percent) 

Support and safety 

.288 

.88 

.67 

.80 

.92 

.94 

Caring staff-student relationships 

.115 

.71 

.39 

.57 

.80 

.84 

Staff-peer relationships 

.143 

.76 

.45 

.63 

.83 

.87 

Professional development needs 

.052 

.51 

.22 

.35 

.62 

.69 

Student health and engagement 

.307 

.89 

.69 

.82 

.93 

.95 

Student delinquency 

.349 

.91 

.73 

.84 

.94 

.96 

Resource provision 

.148 

.77 

.46 

.63 

.84 

.87 


Note: The analytic sample consists of 10,239 teachers in 362 schools with high response rates and 4,331 
teachers in 400 schools with low response rates that administered the survey in 2008/09 or 2009/10. 

a. Reliability predicted from Spearman-Brown formula; recommended value is at least .60. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 


property among schools with low response rates. No response rate group differences in 
schoohlevel reliabilities were apparent on the teacher-reported measures (table E14). 

How stable are the school-level climate measures ? For the student survey, the average 
one- and two-year stability estimates for the subsample with high response rates were 
higher than the estimates for the subsample with low response rates. For the one-year sta- 
bility estimates, the coefficients for the subsample with high response rates were larger 
than those for the subsample with low response rates for five of the six measures (table 
E15). These differences, however, should be interpreted cautiously because there were too 
few schools in the subsample with low response rates that administered the student survey 
in back-to-back years to obtain consistent estimates. Differences in the average one-year 
stability estimates between the full sample (see table 5 in main report) and the subsample 
with high response rates were minimal, though the estimates for the subsample with high 
response rates were slightly higher for all constructs. For example, the average one-year 
stability coefficient for safety and connectedness was .80 in the full sample and .82 in the 
subsample with high response rates. 

For the two-year stability estimates, the coefficients were larger for the subsample with high 
response rates than for the subsample with low response rates for all six measures (table 
E16). For example, the average two-year stability coefficient for caring relationships with 
adults was .48 in the subsample with high response rates and .36 in the subsample with low 
response rates. The results suggest that temporal stability of the student measures is higher 
in the subsample with high response rates than in the subsample with low response rates. 
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Table E15. Stability of student survey measures on the California School Climate, Health, and Learning 
Survey for one-year intervals, by survey year and response rate (correlations) 


Measure 

2004/05 to 
2005/06 

2005/06 to 
2006/07 

2006/07 to 
2007/08 

2007/08 to 
2008/09 

2008/09 to 
2009/10 

2009/10 to 
2010/11 

One-year 

average 

High response rate (at least 60 percent) 

Safety and connectedness 

.83 

.70 

.80 

.81 

.86 

.91 

.82 

Caring relationships with adults 

.25 

.41 

.64 

.66 

.65 

.80 

.57 

Meaningful participation 

.39 

.05 

.77 

.74 

.80 

.81 

.59 

Substance use at school 

.55 

.39 

.08 

.38 

.29 

.60 

.38 

Bullying and discrimination 

.54 

.50 

.51 

.40 

.42 

.65 

.50 

Delinquency 

.71 

.49 

.60 

.57 

.65 

.78 

.63 

Number of survey intervals 

54 

28 

34 

33 

33 

37 

219 

Low response rate (less than 60 percent) 

Safety and connectedness 

.78 

.39 

.71 

.94 

.30 

-.37 

.51 

Caring relationships with adults 

.80 

-.26 

.88 

.83 

.71 

.08 

.53 

Meaningful participation 

.12 

.33 

.21 

.49 

.21 

-.93 

.12 

Substance use at school 

.58 

-.31 

.91 

.63 

.39 

.79 

.48 

Bullying and discrimination 

.70 

.61 

.84 

.71 

.12 

-.84 

.44 

Delinquency 

.56 

-.05 

-.20 

.16 

-.30 

.34 

.09 

Number of survey intervals 

7 

6 

6 

5 

5 

4 

33 


Note: The analytic sample consists of 122 schools (106 schools with high response rates and 16 schools with low response rates) that 
administered the survey in consecutive years. 

Source: 2004/05 to 2010/11 Cal-SCHLS student survey. 


Table E16. Stability of student survey measures on the California School Climate, Health, and Learning 
Survey for two-year intervals, by survey year and response rate (correlations) 


Measure 

2004/05 to 
2006/07 

2005/06 to 
2007/08 

2006/07 to 
2008/09 

2007/08 to 
2009/10 

2008/09 to 
2010/11 

Two-year 

average 

High response rate (at least 60 percent) 

Safety and connectedness 

.64 

.71 

.61 

.76 

.67 

.68 

Caring relationships with adults 

.31 

.55 

.42 

.51 

.53 

.46 

Meaningful participation 

.35 

.56 

.50 

.61 

.63 

.51 

Substance use at school 

.16 

.37 

.34 

.41 

.43 

.34 

Bullying and discrimination 

.52 

.45 

.36 

.44 

.50 

.45 

Delinquency 

.51 

.64 

.52 

.64 

.52 

.57 

Number of survey intervals 

204 

455 

203 

494 

151 

1,507 

Low response rate (less than 60 percent) 

Safety and connectedness 

.64 

.62 

.59 

.64 

.33 

.56 

Caring relationships with adults 

.41 

.34 

.58 

.37 

.08 

.36 

Meaningful participation 

.16 

.36 

.27 

.53 

.26 

.32 

Substance use at school 

.43 

.21 

.33 

.32 

.23 

.30 

Bullying and discrimination 

.35 

.41 

.25 

.17 

.12 

.26 

Delinquency 

.65 

.54 

.51 

.54 

.40 

.53 

Number of survey intervals 

57 

118 

54 

134 

40 

403 


Note: The analytic sample consists of 941 schools (731 schools with high response rates and 210 schools with low response rates) 
that administered the survey in two-year intervals. 

Source: 2004/05 to 2010/11 Cal-SCHLS student survey. 
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For the staff survey, there were too few schools in the subsamples with high and low 
response rates to interpret differences in stability coefficients across groups (table E17). 
However, in comparison with the full sample estimates (see table 5 in main report) for 
the two-year stability estimates, the coefficients for the subsample with high response rates 
were slightly larger than those for the subsample with low response rates for five constructs: 
support and safety, caring staff-student relationships, staff-peer relationships, student 
health and engagement, and resource provision. For example, the two-year stability coef- 
ficient for support and safety was .42 in the full sample, .46 in the subsample with high 
response rates, and .40 in the subsample with low response rates. For professional devel- 
opment needs and student delinquency, the subsample with low response rates had higher 
two-year stability coefficients than the full sample, and the subsample with high response 
rates had lower coefficients. 

Are the school climate measures related to student achievement and discipline infrac- 
tions 1 Correlations between the school climate measures and the four criterion measures 
were stronger in the subsample with high response rates than in the subsample with low 
response rates, particularly for the student survey (tables E18 and E19). The largest differ- 
ences were for caring relationships with adults, meaningful participation, and bullying and 
discrimination. For each of these measures the correlations were substantially stronger in 
the subsample with high response rates than in the subsample with low response rates. 
For example, the correlation between math test scores and caring relationships was .49 in 


Table E17. Stability of staff survey measures on the California School Climate, 
Health, and Learning Survey, by survey interval and response rate (correlations) 



One-year interval 


Two-year 

interval 

Measure 

2008/09 to 
2009/10 

2009/10 to 
2010/11 

One-year 

average 

2008/09 to 
2010/11 

High response rate (at least 50 percent) 

Support and safety 

.79 

.13 

.33 

.46 

Caring staff-student relationships 

.70 

.88 

.78 

.51 

Staff-peer relationships 

.73 

.88 

.80 

.44 

Professional development needs 

-.14 

.76 

.26 

.32 

Student health and engagement 

.74 

.83 

.78 

.79 

Student delinquency 

.88 

.93 

.90 

.62 

Resource provision 

.78 

.06 

.46 

.44 

Number of survey intervals 

14 

11 

25 

55 

Low response rate (less than 50 percent) 

Support and safety 

.68 

.70 

.79 

.40 

Caring staff-student relationships 

.52 

.73 

.63 

.40 

Staff-peer relationships 

.74 

.81 

.78 

.33 

Professional development needs 

.95 

-.93 

.01 

.48 

Student health and engagement 

.79 

.14 

.47 

.59 

Student delinquency 

-.01 

.74 

.37 

.66 

Resource provision 

.56 

.90 

.73 

.41 

Number of survey intervals 

8 

9 

17 

64 


Note: The analytic sample consists of 34 schools (19 schools with high response rates and 15 schools with 
low response rates) that administered the survey in consecutive years and 119 schools (55 schools with high 
response rates and 64 schools with low response rates) that administered the survey in two-year intervals. 

Source: 2008/09 to 2010/11 Cal-SCHLS staff survey. 
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Table E18. School-level correlations of student survey measures on the California 


School Climate, Health, and Learning Survey with student achievement and 

discipline infractions, by high and low response rates 




English language 
arts test scores 

Math test 
scores 


Measure 

Grade 7 

Grade 8 

Grade 7 

Suspensions 

High response rate (at least 60 percent) 

Safety and connectedness 

.68 

.67 

.64 

-.51 

Caring relationships with adults 

.49 

.48 

.49 

-.38 

Meaningful participation 

.55 

.54 

.53 

-.39 

Substance use 

-.48 

-.47 

-.41 

.34 

Bullying and discrimination 

-.23 

-.24 

-.29 

.19 

Delinquency 

-.63 

-.64 

-.57 

.42 

Low response rate (less than 60 percent) 

Safety and connectedness 

.61 

.60 

.46 

-.37 

Caring relationships with adults 

.24 

.26 

.17 

-.17 

Meaningful participation 

.36 

.37 

.26 

-.22 

Substance use 

-.44 

-.39 

-.30 

.22 

Bullying and discrimination 

.01 

-.06 

.02 

.01 

Delinquency 

-.49 

-.53 

-.40 

.24 

Note: The analytic sample consists of 949 middle schools (720 high-response-rate schools and 229 low-re- 
sponse-rate schools) that administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 





Table E19. School-level correlations of staff survey measures on the California 
School Climate, Health, and Learning Survey with student achievement and 

discipline infractions, by high and low response rates 




English language 
arts test scores 

Math test 
scores 


Measure 

Grade 7 

Grade 8 

Grade 7 

Suspensions 

High response rate (at least 60 percent) 

Support and safety 

.48 

.50 

.41 

-.25 

Caring staff-student relationships 

.43 

.45 

.35 

-.23 

Staff-peer relationships 

.28 

.30 

.20 

-.17 

Professional development needs 

-.42 

-.41 

-.34 

.20 

Student health and engagement 

.83 

.82 

.72 

-.47 

Student delinquency 

-.69 

-.70 

-.60 

.46 

Resource provision 

.28 

.31 

.29 

-.19 

Low response rate (less than 60 percent) 

Support and safety 

.43 

.44 

.45 

-.19 

Caring staff-student relationships 

.38 

.38 

.35 

-.14 

Staff-peer relationships 

.26 

.26 

.23 

-.10 

Professional development needs 

-.27 

-.28 

-.27 

.13 

Student health and engagement 

.74 

.74 

.66 

-.39 

Student delinquency 

-.66 

-.67 

-.59 

.44 

Resource provision 

.24 

.23 

.31 

-.10 


Note: The analytic sample consists of 362 high-response-rate schools and 400 low-response-rate schools 
that administered the survey in 2008/09 or 2009/10. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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the subsample with high response rates and .17 in the subsample with low response rates. 
In the subsample with low response rates, predictive validity is not supported for caring 
relationships with adults and bullying and discrimination. 

Similarly, for the staff survey, correlations between the school climate measures and the 
four criterion measures were stronger in the subsample with high response rates than in the 
subsample with low response rates, but the differences were mostly smaller than they were 
for the student survey. The largest difference in correlations between groups was for profes- 
sional development needs and English language arts test scores: —42 in the subsample with 
high response rates compared with -.27 in the subsample with low response rates. 

Overall, greater predictive validity was demonstrated for schools with high survey response 
rates than for schools with low survey response rates. 
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Appendix F. Student survey exploratory factor 
analysis results: Factor loadings from preferred model 


Table FI presents factor loadings for each student survey item estimated through explor- 
atory factor analyses for the preferred model. The model is based on data from a randomly 
generated split-half sample of schools from the full sample of schools that administered the 
student survey of the California School Climate, Health, and Learning Survey in 2008/09 
or 2009/10. 


Table FI. Five-factor exploratory factor analysis model of student survey items on 
the California School Climate, Health, and Learning Survey 


Item 

Description 

1 

2 

3 

4 

5 ! 

11 

1 feel close to people at this school 

.61 

.02 

.14 

-.01 

.01 

12 

1 am happy to be at this school 

.83 

-.06 

-.02 

.02 

-.05 

13 

1 feel like 1 am a part of this school 

.78 

.03 

.03 

-.01 

.02 

14 

Teachers at school treat students fairly 

.36 

.20 

-.16 

-.05 

-.02 

15 

1 feel safe in my school 

.67 

.02 

-.04 

-.11 

-.04 

16 

Adult who really cares about me 

.04 

.71 

.04 

.01 

.07 

17 

. . . who tells me when 1 do a good job 

.02 

.74 

-.02 

.01 

-.03 

18 

. . . who notices when I'm not there 

.00 

.68 

.08 

-.03 

.03 

19 

... who always wants me to do my best 

-.03 

.80 

-.03 

.03 

-.09 

20 

... who listens to me 

.02 

.75 

.00 

-.06 

.00 

21 

... who believes that 1 will be a success 

-.03 

.81 

-.02 

-.03 

-.03 

22 

1 do interesting activities at school 

.28 

.29 

-.01 

.03 

.26 

23 

1 help decide things like activities or rules 

.24 

.35 

-.02 

-.01 

.53 

24 

1 do things at school that make a difference 

.23 

.37 

-.03 

.05 

.46 

55 

Lifetime, been drunk or high at school 

-.04 

-.04 

.76 

.02 

.21 

72 

Past 30 days, smoked cigarettes at school 

-.03 

-.02 

.80 

-.01 

.40 

73 

... drank alcohol at school 

-.03 

.01 

.78 

-.04 

.32 

74 

... smoked marijuana at school 

-.04 

-.02 

.86 

-.05 

.33 

75 

... used other illegal drug at school 

-.02 

.00 

.76 

-.02 

.43 

100 

Past 12 months, been pushed or shoved 

-.03 

.01 

.04 

.67 

-.09 

101 

... been afraid of being beaten up 

-.08 

.05 

-.09 

.67 

-.02 

102 

... been in a physical fight 

-.01 

.02 

.53 

.33 

-.06 

103 

... had rumors spread about you 

.03 

.02 

.00 

.76 

-.03 

104 

... had sexual jokes made to you 

.04 

-.01 

.09 

.69 

-.06 

105 

... been made fun of because of looks 

-.02 

-.01 

-.20 

.82 

-.03 

106 

... had your property stolen or damaged 

.00 

.02 

.12 

.63 

-.01 

107 

... been offered or sold an illegal drug 

.00 

-.01 

.67 

.24 

-.03 

108 

... damaged school property on purpose 

.02 

-.05 

.61 

.30 

-.04 

109 

... carried a gun 

.03 

-.01 

.76 

.24 

.01 

110 

... carried any other weapon 

.03 

.00 

.78 

.23 

-.09 

111 

... been threatened or injured by a weapon 

.00 

.02 

.53 

.46 

.01 

112 

... seen someone carrying a weapon 

.02 

.04 

.56 

.35 

-.11 

113 

... been bullied because of race/ethnicity 

-.02 

-.02 

-.01 

.64 

.25 

114 

... been bullied because of religion 

.00 

-.02 

.03 

.60 

.35 

115 

... been bullied because of gender 

.00 

-.01 

-.01 

.70 

.31 


(continued) 
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Table FI. Five-factor exploratory factor analysis model of student survey items on 
the California School Climate, Health, and Learning Survey (continued) 


Item Description 1 2 3 4 5 


116 

... been bullied because you are LGBT 

-.03 

-.02 

.00 

.69 

.23 

117 

... been bullied because of disability 

-.01 

-.02 

.10 

.61 

.30 

118 

... been bullied for any other reason 

-.05 

.00 

-.15 

.78 

.11 

119 

How safe do you feel at school 

.51 

.04 

-.03 

-.26 

.04 

120 

Past 12 months, been cyber-bullied 

.03 

-.02 

.16 

.49 

.14 


Note: The analytic sample consists of 106,928 students in 464 middle schools that administered the survey 
in 2008/09 or 2009/10. Numbers in bold are factor loadings of at least .30. 

Source: 2008/09 and 2009/10 Cal-SCHLS student survey. 
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Appendix G. Staff survey exploratory factor 
analysis results: Factor loadings from selected model 


Table G1 presents factor loadings for each staff survey item estimated through exploratory 
factor analyses for the selected model. The model is based on data from a randomly gener- 
ated split'half sample of schools from the full sample of schools that administered the Cal- 
ifornia School Climate, Health, and Learning Survey staff survey in 2008/09 or 2009/10. 


Table Gl. Six-factor exploratory factor analysis model of staff survey items on the 
California School Climate, Health, and Learning Survey 


1 Item 

Description 

1 

2 

3 

4 

5 

6 ! 

6 

This school is a supportive place for students to learn 

.73 

.00 

.12 

-.01 

.12 

.19 

7 

... sets high standards for academic performance 

.74 

-.01 

.04 

.01 

.07 

.22 

8 

... promotes academic success for all students 

.87 

-.03 

-.02 

.00 

-.01 

.22 

9 

... emphasizes helping students academically 

.81 

.02 

.01 

-.02 

-.04 

.17 

10 

... provides adequate support services for students 

.64 

-.03 

.06 

.01 

.06 

.05 

11 

... emphasizes teaching lessons in relevant ways 

.77 

.05 

-.02 

.01 

-.02 

.10 

12 

... is a supportive and inviting place for staff to work 

.52 

.06 

.56 

.01 

-.12 

.17 

13 

... promotes trust and collegiality among staff 

.47 

.14 

.57 

.01 

-.16 

.14 

14 

... provides resources for staff to do their jobs well 

.82 

-.20 

.03 

-.03 

.00 

.16 

15 

... provides resources to work with special education 








(IEP) students 

.84 

-.19 

-.01 

-.06 

.01 

.12 

16 

... provides opportunities for student decision-making 

.69 

.00 

-.04 

.05 

.04 

-.02 

17 

... gives students class discussion opportunities 

.72 

.10 

.01 

-.04 

.01 

-.03 

18 

... gives students extracurricular activity opportunities 

.68 

.00 

.00 

-.01 

.07 

-.01 

19 

... gives students opportunities to make a difference 

.68 

-.01 

.00 

.00 

.10 

.01 

20 

... encourages students to enroll in rigorous courses 

.70 

.03 

-.01 

-.01 

.06 

-.01 

21 

... emphasizes culturally relevant instruction 

.81 

.02 

-.12 

-.01 

-.05 

-.16 

22 

... has staff examine their cultural biases through P.D. 

.75 

.06 

-.12 

.00 

-.11 

-.21 

23 

... prioritizes closing the racial achievement gap 

.69 

.12 

-.03 

.00 

-.13 

-.23 

24 

... fosters appreciation for student diversity 

.77 

.15 

.04 

-.01 

.01 

-.27 

25 

... emphasizes showing respect for student cultures 

.76 

.16 

.05 

-.02 

-.01 

-.29 

26 

... clearly communicates rules to students 

.43 

-.01 

.58 

.00 

.09 

-.18 

27 

... handles discipline problems fairly 

.37 

.02 

.71 

.01 

.11 

-.25 

28 

... effectively handles discipline and behavior problems 

.38 

-.02 

.68 

.01 

.13 

-.21 

29 

... is a safe place for students 

.49 

-.04 

.42 

.01 

.28 

-.03 

30 

... is a safe place for staff 

.50 

-.05 

.45 

.01 

.23 

-.01 

31 

... is welcoming to and facilitates parent involvement 

.63 

.05 

.21 

.01 

.09 

.00 

32 

... has clean and well-maintained facilities 

.45 

.02 

.19 

.05 

.14 

.03 

33 

Adults in this school really care about every student 

.04 

.87 

-.04 

.00 

.03 

.16 

34 

... acknowledge and pay attention to students 

.03 

.91 

-.05 

.00 

.03 

.17 

35 

... want every student to do their best 

.07 

.83 

.01 

.01 

.02 

.06 

36 

... listen to what students have to say 

.07 

.87 

-.04 

.00 

.05 

.00 

37 

... believe that every student can be a success 

.11 

.82 

-.03 

.02 

.06 

-.02 

38 

... treat all students fairly 

.00 

.90 

.07 

-.01 

.11 

-.18 

39 

... treat every student with respect 

-.02 

.90 

.08 

.00 

.11 

-.18 

40 

... have close professional relationships with each other 

-.02 

.68 

.42 

-.03 

-.09 

.04 


(continued) 
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Table Gl. Six-factor exploratory factor analysis model of staff survey items on the 
California School Climate, Health, and Learning Survey (continued) 


1 Item 

Description 

1 

2 

3 

4 

5 

6 ! 

41 

... support and treat each other with respect 

-.07 

.73 

.49 

-.03 

-.08 

.03 

42 

... feel a responsibility to improve this school 

.08 

.69 

.30 

.01 

-.08 

.01 

43 

1 need training in meeting academic standards 

.11 

.03 

-.07 

.88 

.03 

-.08 

44 

... evidence-based methods of instruction 

.03 

.04 

-.02 

.85 

.07 

-.06 

45 

... positive behavioral supports/classroom management 

.07 

.02 

-.07 

.78 

-.09 

-.06 

46 

... working with diverse racial and cultural groups 

-.02 

-.03 

.06 

.88 

.01 

.13 

47 

... culturally relevant pedagogy for students 

-.08 

-.02 

.09 

.86 

-.01 

.11 

48 

... serving English language learners 

-.02 

.02 

.08 

.82 

.00 

.02 

49 

... closing the achievement gap 

-.02 

-.01 

.05 

.81 

-.01 

-.01 

50 

... serving special education students 

-.08 

.01 

.03 

.77 

.02 

.01 

51 

... meeting the social and emotional needs of students 

.00 

-.03 

-.03 

.81 

-.03 

.03 

52 

... creating a positive school environment 

.00 

-.09 

-.19 

.77 

-.05 

.00 

53 

Students at this school are healthy and physically fit 

.02 

.14 

-.07 

.05 

.47 

.48 

54 

... arrive at school alert and rested 

.02 

.16 

-.05 

.02 

.54 

.56 

55 

... are motivated to learn 

.04 

.11 

.06 

-.01 

.50 

.58 

56 

... are well-behaved 

.00 

.05 

.16 

-.03 

.53 

.50 

57 

Student alcohol and drug use is not a problem at school 

.00 

.03 

-.09 

.03 

.82 

-.17 

58 

Student tobacco use... 

-.05 

.04 

-.09 

.03 

.79 

-.17 

59 

Harassment and bullying among students... 

.08 

.05 

.04 

-.03 

.66 

-.07 

60 

Physical fighting between students... 

.01 

.01 

.05 

-.03 

.78 

.01 

61 

Disruptive student behavior... 

.06 

-.04 

.13 

-.02 

.71 

.15 

62 

Racial conflict among students... 

.05 

.05 

.02 

-.06 

.60 

-.08 

63 

Student depression or mental health... 

.03 

.06 

-.02 

-.04 

.55 

-.04 

64 

Lack of respect for staff by students... 

.03 

.00 

.21 

-.01 

.67 

.17 

65 

Cutting classes or being truant... 

-.01 

-.01 

.08 

-.03 

.73 

.11 

66 

Gang-related activity... 

.01 

-.05 

-.02 

-.03 

.80 

.06 

67 

Weapons possession... 

-.04 

-.01 

.00 

-.03 

.81 

.00 

68 

Vandalism... 

.01 

-.04 

.01 

.01 

.81 

.03 

69 

Theft... 

-.02 

.02 

.00 

.01 

.76 

.00 


Note: The analytic sample consists of 7,079 teachers in 393 middle schools that administered the survey in 
2008/09 or 2009/10. Numbers in bold are factor loadings of at least .30. 

Source: 2008/09 and 2009/10 Cal-SCHLS staff survey. 
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Notes 


1. A supplemental module of the CahSCHLS that was designed for high schools par- 
ticipating in the Safe and Supportive Schools Initiative includes questions about aca- 
demic rigor, respectful relationships and cultural sensitivity, the relevance of classroom 
lessons, discipline and enforcement of rules, and the quality of facilities maintenance. 

2. Since the student survey was administered to grade 7 students only, it is unclear 
whether a sample of 100 students across all grades in middle schools (that is, 33-34 
students in each grade) would produce the same reliability estimates. 

3. Because the schoohlevel reliability estimates represent the proportion of total vari- 
ance that is not due to measurement error or sampling error, the estimated number 
of respondents needed per school is an upper bound estimate, assuming an infinite 
school population size. 

4. For this study, middle schools include all traditional schools with grade 6-8 and grade 
7-8 configurations only. K-8 schools or schools with other grade configurations were 
not included in the analytic sample. 

5. This cutoff is a commonly accepted rule of thumb in assessing goodness of model fit 
using RMSEA. There are arguments that a smaller threshold may be more effective in 
correctly rejecting misspecified models when the sample size is large (Chen, Curran, 
Bollen, Kirby, & Paxton, 2008). 

6. SRMR is not available in Mplus when item thresholds are estimated, as they were in 
the CFA models. 

7. The average variance explained by the latent factor on observed indicators is the 
average of the squared factor loadings. The shared variance explained by latent factors 
is the squared correlations between latent factors in the CFA models described above. 

8. The aggregated constructs are based on the responses of different students across years 
(grade 7 students in 2007, grade 7 students in 2009). 

9. It is possible to determine whether schools with lower response rates differ from other 
schools in school climate characteristics by aggregating the school climate measures 
to the school level and making direct comparisons. However, these comparisons can 
only reveal if the respondents who completed the surveys in schools with high and low 
response rates report similar levels of school climate. The comparisons do not provide 
information on the differences between respondents and nonrespondents in schools. 
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